Getting started with neuralfields
¶
This pages explains how to install neuralfields
via the package management system poetry
, how the CI/CD pipelines are set up using poe
tasks, and how pre-commit
hooks are configures to keep the commits clean.
Installation¶
Installation of poetry
¶
This project is managed by poetry, a Python packaging and dependency management tool. Therefore, poetry
must be installed first.
Please have a look at the official poetry documentation on how to install poetry
on different platforms and under different conditions.
Installation of neuralfields
¶
The installation of this project is quite straightforward. Simply go to the project's directory, and run
poetry install
No project development intended?
If you don't need any development setup, you can pass the --no-dev
flag to skip the development dependencies.
Computer says no...
Please find a collection of known failure cases below, and feel free to report more.
Symptom | Hint |
---|---|
Something is wrong with my poetry environment | Delete the .venv folder and recreate the virtual environment. |
Dependency Management & Packaging¶
As mentioned in the Installation section, poetry is employed to keep the dependencies of different projects from interfering with each other. By running poetry install
in the project's root folder, a separate virtual environment is created into which all dependencies are installed automatically (typically takes a few seconds to minutes). This is similar to running pip install -r requirements.txt
in an isolated virtual environment.
Poe Task Runner¶
This project defines so-called tasks using poe which are executed in commit hooks as well as the CI/CD pipeline. These tasks are essentially a sequence of commands, and can also be executed locally in the terminal by running
poetry run poe <task_name>
Available tasks
To get a list of available tasks, execute poetry run poe --help
Poe the Poet - A task runner that works well with poetry.
version 0.17.1
USAGE
poe [-h] [-v | -q] [--root PATH] [--ansi | --no-ansi] task [task arguments]
GLOBAL OPTIONS
-h, --help Show this help page and exit
--version Print the version and exit
-v, --verbose Increase command output (repeatable)
-q, --quiet Decrease command output (repeatable)
-d, --dry-run Print the task contents but don't actually run it
--root PATH Specify where to find the pyproject.toml
--ansi Force enable ANSI output
--no-ansi Force disable ANSI output
CONFIGURED TASKS
bump-version-tag Bump version. This creates a new git tag based on the desired version part. Note that this task does not
actually push the tag. You can do this manually, e.g. by running 'poe push-latest-version-tag'.
part Part of version being bumped. Allowed values: patch, minor, major.
--release Wether this is a release. Then, the tag will be annotated.
clean Clean up all temporary files.
format Format or check Python files with black, isort, pyupgrade & autoflake.
--check If true, only check if the files are formatted but do not format them.
files List of files (optional).
lint Lint Python files with mypy, pylint, and bandit. The reports are stored in the given directory.
files List of files or directories (optional).
--reportdir Diretory to write the linters' reports to (optional).
test Run the project's tests using pytest (with --exitfirst). Then compute the test coverage and compile it to html.
docs Build the docs (needs completed test task).
deploy-docs Deploy the docs (needs completed docs task).
--alias Version alias.
--push Wether to push the docs to GitHub pages.
--version-postfix Information appended to version (optional).
deploy-package Deploy package to PyPI (no --repository PRIVATE_REPO is needed).
--username Repository user name.
--password Repository password / access token.
push-latest-version-tag Push the latest version tag.
release Make a new (stable) release. This will test the package, create a new tag based on the version, build and
deploy the docs, and finally push the new tag to remote.
part Release type. Allowed values: patch, minor, major.
--password The repository password / access token.
--username The repository user name.
Git Hooks¶
This project uses pre-commit hooks to automatically check for common formatting issues. The hooks are executed before every commit, but can be disabled by adding --no-verify
when committing, e.g. git commit . -m "Fixed something" --no-verify
.
Installation of pre-commit
After you cloned this project and plan to develop in it, don't forget to install these hooks via
poetry run pre-commit install
Available pre-commit hooks
The pre-commit hooks are configured in the .pre-commit-config.yaml
file as follows
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.4.0
hooks:
- id: check-added-large-files
args: [--maxkb=5000] # limit to 5MB per file
- id: check-toml
- id: check-yaml
args: [--unsafe] # instead of loading the files, simply parse them for syntax
- id: destroyed-symlinks
- id: detect-private-key
- id: end-of-file-fixer
- id: fix-byte-order-marker
- id: mixed-line-ending
- id: name-tests-test
args: [--pytest-test-first]
- id: trailing-whitespace
- repo: https://github.com/psf/black
rev: 23.1.0
hooks:
- id: black
- repo: https://github.com/pycqa/isort
rev: 5.12.0
hooks:
- id: isort
GitHub Actions¶
There are basic CI, CD, and Release pipelines which are executed as GitHub actions workflow on pushing changes or opening pull requests.
Available workflows
name: Continuous Integration
on: [pull_request, push, workflow_dispatch]
defaults:
run:
shell: bash
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
cancel-in-progress: true
jobs:
ci:
name: CI
strategy:
matrix:
os: [ubuntu-latest] # [ubuntu-latest, macos-latest, windows-latest]
python-version: [ "3.8", "3.10", "3.11" ]
runs-on: ${{ matrix.os }}
env:
os: ${{ matrix.os }}
python: ${{ matrix.python-version }}
steps:
- name: Install prerequisites
env:
DEBIAN_FRONTEND: noninteractive
run: |
sudo apt-get update
sudo apt-get install --yes curl gcc git
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Set up poetry
run: |
curl -sSL https://install.python-poetry.org | python3 - --force
echo "$HOME/.local/bin" >> $GITHUB_PATH
- name: Checkout repository
uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Configure git
run: |
git config user.name "${GITHUB_ACTOR}"
git config user.email "${GITHUB_ACTOR}@users.noreply.github.com"
- name: Install dependencies
run: poetry install
- name: Install and run pre-commit hooks
env:
SKIP: no-commit-to-branch
run: |
poetry run pre-commit install
poetry run pre-commit run --all-files --show-diff-on-failure
- name: Run tests
run: poetry run poe test
- name: Code Coverage Summary
uses: irongut/CodeCoverageSummary@v1.3.0
with:
filename: coverage.xml
format: markdown
output: both
- name: Build docs
run: poetry run poe docs
- name: Deploy temporary docs
if: github.event_name == 'pull_request'
run: |
poetry run poe deploy-docs \
--push \
--alias pr-${{ github.event.number }} \
--version-postfix pr-${{ github.event.number }}
- name: Write PR note
if: github.event_name == 'pull_request'
run: |
cat <<EOT>> pr_ci_note.md
[temporary docs](https://github.com/famura/neuralfields/pr-${{ github.event.number }}").
[tests](https://github.com/famura/neuralfields/pr-${{ github.event.number }}/exported/tests/report.html)
[coverage](https://github.com/famura/neuralfields/pr-${{ github.event.number }}/exported/coverage/report.html)
EOT
- name: Add PR note
if: github.event_name == 'pull_request'
uses: marocchino/sticky-pull-request-comment@v2.3.1
with:
path: pr_ci_note.md
name: Continuous Deployment
on:
push:
branches: [main]
workflow_dispatch:
defaults:
run:
shell: bash
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
cancel-in-progress: true
jobs:
cd:
name: CD
if: github.repository == 'famura/neuralfields'
runs-on: ubuntu-latest
permissions: # https://docs.github.com/en/actions/security-guides/automatic-token-authentication
contents: write # to publish the docs to gh-pages
env:
os: ubuntu-latest
python: "3.10"
steps:
- name: Install prerequisites
env:
DEBIAN_FRONTEND: noninteractive
run: |
sudo apt-get update
sudo apt-get install --yes curl gcc git
- name: Set up Python 3.10
uses: actions/setup-python@v4
with:
python-version: "3.10"
- name: Set up poetry
run: |
curl -sSL https://install.python-poetry.org | python3 - --force
echo "$HOME/.local/bin" >> $GITHUB_PATH
- name: Checkout repository
uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Configure git
run: |
git config user.name "${GITHUB_ACTOR}"
git config user.email "${GITHUB_ACTOR}@users.noreply.github.com"
- name: Install dependencies
run: poetry install
- name: Bump patch version
run: poetry run poe bump-version-tag patch
- name: Deploy docs
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
poetry run poe test
poetry run poe docs
poetry run poe deploy-docs --push --alias latest
- name: Deploy package
run: |
poetry run poe deploy-package --username ${{ secrets.PYPI_USER }} --password ${{ secrets.PYPI_TOKEN }}
- name: Push version tag
run: poetry run poe push-latest-version-tag
release_draft:
name: Update release notes
runs-on: ubuntu-latest
permissions: # https://docs.github.com/en/actions/security-guides/automatic-token-authentication
contents: write
steps:
- uses: release-drafter/release-drafter@v5.22.0
with:
config-name: release_drafter.yaml
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
name: Release
on:
workflow_dispatch:
inputs:
bumped-version-part:
description: "The version part to bump."
type: choice
options:
- major
- minor
- patch
default: "minor"
required: true
defaults:
run:
shell: bash
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
cancel-in-progress: true
jobs:
release:
name: Release
if: github.repository == 'famura/neuralfields'
runs-on: ubuntu-latest
permissions: # https://docs.github.com/en/actions/security-guides/automatic-token-authentication
contents: write # to publish the docs to gh-pages
env:
os: ubuntu-latest
python: "3.10"
steps:
- name: Install prerequisites
env:
DEBIAN_FRONTEND: noninteractive
run: |
sudo apt-get update
sudo apt-get install --yes curl gcc git
- name: Set up Python 3.10
uses: actions/setup-python@v4
with:
python-version: "3.10"
- name: Set up poetry
run: |
curl -sSL https://install.python-poetry.org | python3 - --force
echo "$HOME/.local/bin" >> $GITHUB_PATH
- name: Checkout repository
uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Configure git
run: |
git config user.name "${GITHUB_ACTOR}"
git config user.email "${GITHUB_ACTOR}@users.noreply.github.com"
- name: Install dependencies
run: poetry install
- name: Publish to PyPI
run: |
poetry run poe release ${{ github.event.inputs.bumped-version-part }} \
--username ${{ secrets.PYPI_USER }} \
--password ${{ secrets.PYPI_TOKEN }}