Skip to content

Tooling rewrite

skwerl requested to merge tooling-rewrite into main

This MR is a total rewrite of the tooling in the main development repo. The rewrite changes it to be a mix project which provides various mix tasks to do common or repetitive development actions.

So far, the tasks provided are:

  • mix dev.setup
    • downloads and prepares all of the repos in config.toml
  • mix dev.update.gitlab
    • regenerates .gitlab-ci.yml from an eex template using version specs in config.toml
    • currently only updates the file in the dev repo, but it is planned to update it in the child repos too, including making a commit

Planned tasks are:

  • mix dev.update.asdf
    • update the .tool-versions files across all repos
    • still need to decide whether this will use the max or min supported version based on config.toml
  • mix dev.update.deps
    • update each repo to use the newest versions of our projects
    • for bot repos only, this will also update all other deps
    • can add our deps with --add flag
  • mix dev.update.secrets
    • update each repo which needs secrets with a secret specified in config.secret.toml which is not included in the repo
  • mix dev.update.template
    • regenerate a given file in each repo from its template
  • mix dev.local
    • changes a dependency to one of our projects to use :path instead of :git to see local changes
    • flags the repo as having local deps, which will prevent commits using a git hook
    • can also change it back to normal
  • mix dev.new
    • create a new repo based on one of several templates
    • types are bot, module and library
    • core repos (like beholder) must still be made manually, but this does not happen often
  • mix dev.release
    • cut a new release of the given repo
    • adds a new git tag and updates the @version in mix.exs
  • mix dev.sync
    • tries to fetch remote work and auto-rebase local work on top
    • should fail safely
  • mix dev
    • should print a list a of subtasks+shortdocs and tell the user to call one of those instead

Parts of this MR in need of testing:

  • test mix dev.setup
    • downloads and builds each project correctly
      • poll, replaces, roles, and scry are expected to fail since they have empty repositories
    • test with and without asdf installed (this was a pain point for me)
    • make sure the versions it builds with matches the versions in .tool-versions if you have asdf installed
      • this is currently known broken; i am working on a fix but its not commited because of Reasons
      • test this by going into the dir and running mix compile manually
        • if the version was right, it will have nothing to do
    • a second run should take far less time than before but produce the same errors and warnings
    • please report timings for both a first and second run
      • you can rm -rf repos/ to remove the repos to get "first" run timing
      • time mix dev.setup
  • test mix dev.update.gitlab
    • it should generate a valid yml on your machine
    • it should have one lint_bash job, one docs job, and the following jobs repeated for each valid combination or erlang, elixir, and alpine:
      • deps
      • build
      • credo
      • format
      • dialyzer
      • test
    • right now, the only invalid combination of elixir and erlang in our matrix is elixir 1.12.3 and erlang 25.2.3
      • its possible you will see versions newer than this, but you should not see older versions
    • some combos of elixir/erlang will show up more than once since we support multiple alpine versions

Parts of this MR in need of review:

  • is the ux good?
  • do the planned tasks cover the things you see yourself needing to do?
  • is it easy to understand what each step does?
    • would it be easier to understand if the logic to skip steps was moved into Step.do_steps/3?
  • is there value in providing a struct to make the structure explicit?
  • is it easy to understand how the setup mix task works?
    • does the idiom of composing steps into tasks make sense? is there a better one?
  • is it easy to understand how the update gitlab mix task works?

Work still to do:

  • think about how to handle repos which have work in progress
    • on a branch other than main
    • modified files
    • local commits
    • unfetched remote commits
    • in the middle of a rebase
    • other states?
  • add moduledocs to each step
    • clone
    • compile
    • credo
    • deps
    • dialyzer
    • docs
    • ready
    • secrets
    • tasks
    • tests
  • add moduledocs to other modules
    • development/gitlab
    • helpers
    • step
  • add longdocs to tasks
    • dev.setup
    • dev.update.gitlab
    • consider removing -h/--help flag in favor of longdocs
  • add files and templates for planned tasks
    • .tool-versions.eex
    • .gitignore.eex
    • mix.exs.eex
    • README.md.eex
    • .credo.exs
    • .formatter.exs
    • config/
      • config.exs
      • config_with_nostrum.exs.eex
    • lib/
      • module.ex.eex
      • bot.ex.eex
      • library.ex.eex
      • application.ex.eex
    • test/
      • test_helper.exs
      • test.exs.eex
  • add git hooks
    • pre-commit.sh (delegates to pre-commit.d/*)
      • 00-localdeps.sh
        • reject commit if .localdeps exists and mix.exs or mix.lock is staged
      • credo.sh
        • reject commit if credo fails
      • dialyzer.sh
        • reject commit if dialyzer fails
      • format.sh
        • could either dumbly reject unformatted changes, or reformat them on the fly
        • ref: https://prettier.io/docs/en/precommit.html#option-6-shell-script
        • reformatting is risky since it may incorrectly handle partially staged files
        • dumbly rejecting also doesnt really work on partially staged files since it rejects staged chunks if an unstaged one is unformatted
        • need to do more research here
  • start planning work on other mix tasks
    • open an issue for each
      • goal of task
      • steps to do task, in order
      • planned cli
      • list of concerns/considerations

Re: other issues:

  • fixes #3 (moved)
    • the unlinking script is not needed with the .gitlab-ci.yml generated by the new mix task
  • obsoletes #1
    • #1 should be closed and any tasks which are not covered by this MR should be opened as new issuesz

Anything not completed in this MR will be split out into other issues

Edited by skwerl

Merge request reports

Loading