Skip to content

Getting started with neuralfields

This pages explains how to install neuralfields via the package management system poetry, how the CI/CD pipelines are set up using poe tasks, and how pre-commit hooks are configures to keep the commits clean.

Installation

Installation of poetry

This project is managed by poetry, a Python packaging and dependency management tool. Therefore, poetry must be installed first.

Please have a look at the official poetry documentation on how to install poetry on different platforms and under different conditions.

Installation of neuralfields

The installation of this project is quite straightforward. Simply go to the project's directory, and run

poetry install

No project development intended?

If you don't need any development setup, you can pass the --no-dev flag to skip the development dependencies.

Computer says no...

Please find a collection of known failure cases below, and feel free to report more.

Symptom Hint
Something is wrong with my poetry environment Delete the .venv folder and recreate the virtual environment.

Dependency Management & Packaging

As mentioned in the Installation section, poetry is employed to keep the dependencies of different projects from interfering with each other. By running poetry install in the project's root folder, a separate virtual environment is created into which all dependencies are installed automatically (typically takes a few seconds to minutes). This is similar to running pip install -r requirements.txt in an isolated virtual environment.

Poe Task Runner

This project defines so-called tasks using poe which are executed in commit hooks as well as the CI/CD pipeline. These tasks are essentially a sequence of commands, and can also be executed locally in the terminal by running

poetry run poe <task_name>
Available tasks

To get a list of available tasks, execute poetry run poe --help

Poe the Poet - A task runner that works well with poetry.
version 0.17.1

USAGE
  poe [-h] [-v | -q] [--root PATH] [--ansi | --no-ansi] task [task arguments]

GLOBAL OPTIONS
  -h, --help     Show this help page and exit
  --version      Print the version and exit
  -v, --verbose  Increase command output (repeatable)
  -q, --quiet    Decrease command output (repeatable)
  -d, --dry-run  Print the task contents but don't actually run it
  --root PATH    Specify where to find the pyproject.toml
  --ansi         Force enable ANSI output
  --no-ansi      Force disable ANSI output

CONFIGURED TASKS
  bump-version-tag         Bump version. This creates a new git tag based on the desired version part. Note that this task does not
actually push the tag. You can do this manually, e.g. by running 'poe push-latest-version-tag'.
    part                   Part of version being bumped. Allowed values: patch, minor, major.
    --release              Wether this is a release. Then, the tag will be annotated.
  clean                    Clean up all temporary files.
  format                   Format or check Python files with black, isort, pyupgrade & autoflake.
    --check                If true, only check if the files are formatted but do not format them.
    files                  List of files (optional).
  lint                     Lint Python files with mypy, pylint, and bandit. The reports are stored in the given directory.
    files                  List of files or directories (optional).
    --reportdir            Diretory to write the linters' reports to (optional).
  test                     Run the project's tests using pytest (with --exitfirst). Then compute the test coverage and compile it to html.
  docs                     Build the docs (needs completed test task).
  deploy-docs              Deploy the docs (needs completed docs task).
    --alias                Version alias.
    --push                 Wether to push the docs to GitHub pages.
    --version-postfix      Information appended to version (optional).
  deploy-package           Deploy package to PyPI (no --repository PRIVATE_REPO is needed).
    --username             Repository user name.
    --password             Repository password / access token.
  push-latest-version-tag  Push the latest version tag.
  release                  Make a new (stable) release. This will test the package, create a new tag based on the version, build and
deploy the docs, and finally push the new tag to remote.
    part                   Release type. Allowed values: patch, minor, major.
    --password             The repository password / access token.
    --username             The repository user name.

Git Hooks

This project uses pre-commit hooks to automatically check for common formatting issues. The hooks are executed before every commit, but can be disabled by adding --no-verify when committing, e.g. git commit . -m "Fixed something" --no-verify.

Installation of pre-commit

After you cloned this project and plan to develop in it, don't forget to install these hooks via

poetry run pre-commit install
Available pre-commit hooks

The pre-commit hooks are configured in the .pre-commit-config.yaml file as follows

repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
  rev: v4.4.0
  hooks:
  - id: check-added-large-files
    args: [--maxkb=5000] # limit to 5MB per file
  - id: check-toml
  - id: check-yaml
    args: [--unsafe] # instead of loading the files, simply parse them for syntax
  - id: destroyed-symlinks
  - id: detect-private-key
  - id: end-of-file-fixer
  - id: fix-byte-order-marker
  - id: mixed-line-ending
  - id: name-tests-test
    args: [--pytest-test-first]
  - id: trailing-whitespace
- repo: https://github.com/psf/black
  rev: 23.1.0
  hooks:
  - id: black
- repo: https://github.com/pycqa/isort
  rev: 5.12.0
  hooks:
  - id: isort

GitHub Actions

There are basic CI, CD, and Release pipelines which are executed as GitHub actions workflow on pushing changes or opening pull requests.

Available workflows
name: Continuous Integration

on: [pull_request, push, workflow_dispatch]

defaults:
  run:
    shell: bash

concurrency:
  group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
  cancel-in-progress: true

jobs:
  ci:
    name: CI
    strategy:
      matrix:
        os: [ubuntu-latest] # [ubuntu-latest, macos-latest, windows-latest]
        python-version: [ "3.8", "3.10", "3.11" ]
    runs-on: ${{ matrix.os }}
    env:
      os: ${{ matrix.os }}
      python: ${{ matrix.python-version }}

    steps:
      - name: Install prerequisites
        env:
          DEBIAN_FRONTEND: noninteractive
        run: |
          sudo apt-get update
          sudo apt-get install --yes curl gcc git

      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v4
        with:
          python-version: ${{ matrix.python-version }}

      - name: Set up poetry
        run: |
          curl -sSL https://install.python-poetry.org | python3 - --force
          echo "$HOME/.local/bin" >> $GITHUB_PATH

      - name: Checkout repository
        uses: actions/checkout@v3
        with:
          fetch-depth: 0

      - name: Configure git
        run: |
          git config user.name "${GITHUB_ACTOR}"
          git config user.email "${GITHUB_ACTOR}@users.noreply.github.com"

      - name: Install dependencies
        run: poetry install

      - name: Install and run pre-commit hooks
        env:
          SKIP: no-commit-to-branch
        run: |
          poetry run pre-commit install
          poetry run pre-commit run --all-files --show-diff-on-failure

      - name: Run tests
        run: poetry run poe test

      - name: Code Coverage Summary
        uses: irongut/CodeCoverageSummary@v1.3.0
        with:
          filename: coverage.xml
          format: markdown
          output: both

      - name: Build docs
        run: poetry run poe docs

      - name: Deploy temporary docs
        if: github.event_name == 'pull_request'
        run: |
          poetry run poe deploy-docs \
            --push \
            --alias pr-${{ github.event.number }} \
            --version-postfix pr-${{ github.event.number }}

      - name: Write PR note
        if: github.event_name == 'pull_request'
        run: |
          cat <<EOT>> pr_ci_note.md

          [temporary docs](https://github.com/famura/neuralfields/pr-${{ github.event.number }}").
          [tests](https://github.com/famura/neuralfields/pr-${{ github.event.number }}/exported/tests/report.html)
          [coverage](https://github.com/famura/neuralfields/pr-${{ github.event.number }}/exported/coverage/report.html)

          EOT

      - name: Add PR note
        if: github.event_name == 'pull_request'
        uses: marocchino/sticky-pull-request-comment@v2.3.1
        with:
          path: pr_ci_note.md
name: Continuous Deployment

on:
  push:
    branches: [main]
  workflow_dispatch:

defaults:
  run:
    shell: bash

concurrency:
  group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
  cancel-in-progress: true

jobs:
  cd:
    name: CD
    if: github.repository == 'famura/neuralfields'
    runs-on: ubuntu-latest
    permissions: # https://docs.github.com/en/actions/security-guides/automatic-token-authentication
      contents: write # to publish the docs to gh-pages
    env:
      os: ubuntu-latest
      python: "3.10"

    steps:
      - name: Install prerequisites
        env:
          DEBIAN_FRONTEND: noninteractive
        run: |
          sudo apt-get update
          sudo apt-get install --yes curl gcc git

      - name: Set up Python 3.10
        uses: actions/setup-python@v4
        with:
          python-version: "3.10"

      - name: Set up poetry
        run: |
          curl -sSL https://install.python-poetry.org | python3 - --force
          echo "$HOME/.local/bin" >> $GITHUB_PATH

      - name: Checkout repository
        uses: actions/checkout@v3
        with:
          fetch-depth: 0

      - name: Configure git
        run: |
          git config user.name "${GITHUB_ACTOR}"
          git config user.email "${GITHUB_ACTOR}@users.noreply.github.com"

      - name: Install dependencies
        run: poetry install

      - name: Bump patch version
        run: poetry run poe bump-version-tag patch

      - name: Deploy docs
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          poetry run poe test
          poetry run poe docs
          poetry run poe deploy-docs --push --alias latest

      - name: Deploy package
        run: |
          poetry run poe deploy-package --username ${{ secrets.PYPI_USER }} --password ${{ secrets.PYPI_TOKEN }}

      - name: Push version tag
        run: poetry run poe push-latest-version-tag

  release_draft:
    name: Update release notes
    runs-on: ubuntu-latest
    permissions: # https://docs.github.com/en/actions/security-guides/automatic-token-authentication
      contents: write

    steps:
      - uses: release-drafter/release-drafter@v5.22.0
        with:
          config-name: release_drafter.yaml
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
name: Release

on:
  workflow_dispatch:
    inputs:
      bumped-version-part:
        description: "The version part to bump."
        type: choice
        options:
          - major
          - minor
          - patch
        default: "minor"
        required: true

defaults:
  run:
    shell: bash

concurrency:
  group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
  cancel-in-progress: true

jobs:
  release:
    name: Release
    if: github.repository == 'famura/neuralfields'
    runs-on: ubuntu-latest
    permissions: # https://docs.github.com/en/actions/security-guides/automatic-token-authentication
      contents: write # to publish the docs to gh-pages
    env:
      os: ubuntu-latest
      python: "3.10"

    steps:
      - name: Install prerequisites
        env:
          DEBIAN_FRONTEND: noninteractive
        run: |
          sudo apt-get update
          sudo apt-get install --yes curl gcc git

      - name: Set up Python 3.10
        uses: actions/setup-python@v4
        with:
          python-version: "3.10"

      - name: Set up poetry
        run: |
          curl -sSL https://install.python-poetry.org | python3 - --force
          echo "$HOME/.local/bin" >> $GITHUB_PATH

      - name: Checkout repository
        uses: actions/checkout@v3
        with:
          fetch-depth: 0

      - name: Configure git
        run: |
          git config user.name "${GITHUB_ACTOR}"
          git config user.email "${GITHUB_ACTOR}@users.noreply.github.com"

      - name: Install dependencies
        run: poetry install

      - name: Publish to PyPI
        run: |
          poetry run poe release ${{ github.event.inputs.bumped-version-part }} \
            --username ${{ secrets.PYPI_USER }} \
            --password ${{ secrets.PYPI_TOKEN }}