Tooling rewrite
This MR is a total rewrite of the tooling in the main development repo. The rewrite changes it to be a mix project which provides various mix tasks to do common or repetitive development actions.
So far, the tasks provided are:
-
mix dev.setup
- downloads and prepares all of the repos in
config.toml
- downloads and prepares all of the repos in
-
mix dev.update.gitlab
- regenerates .gitlab-ci.yml from an eex template using version specs in
config.toml
- currently only updates the file in the dev repo, but it is planned to update it in the child repos too, including making a commit
- regenerates .gitlab-ci.yml from an eex template using version specs in
Planned tasks are:
-
mix dev.update.asdf
- update the
.tool-versions
files across all repos - still need to decide whether this will use the max or min supported version based on
config.toml
- update the
-
mix dev.update.deps
- update each repo to use the newest versions of our projects
- for bot repos only, this will also update all other deps
- can add our deps with
--add
flag
-
mix dev.update.secrets
- update each repo which needs secrets with a secret specified in
config.secret.toml
which is not included in the repo
- update each repo which needs secrets with a secret specified in
-
mix dev.update.template
- regenerate a given file in each repo from its template
-
mix dev.local
- changes a dependency to one of our projects to use
:path
instead of:git
to see local changes - flags the repo as having local deps, which will prevent commits using a git hook
- can also change it back to normal
- changes a dependency to one of our projects to use
-
mix dev.new
- create a new repo based on one of several templates
- types are
bot
,module
andlibrary
- core repos (like
beholder
) must still be made manually, but this does not happen often
-
mix dev.release
- cut a new release of the given repo
- adds a new git tag and updates the
@version
inmix.exs
-
mix dev.sync
- tries to fetch remote work and auto-rebase local work on top
- should fail safely
-
mix dev
- should print a list a of subtasks+shortdocs and tell the user to call one of those instead
Parts of this MR in need of testing:
- test
mix dev.setup
- downloads and builds each project correctly
- poll, replaces, roles, and scry are expected to fail since they have empty repositories
- test with and without asdf installed (this was a pain point for me)
- make sure the versions it builds with matches the versions in .tool-versions if you have asdf installed
- this is currently known broken; i am working on a fix but its not commited because of Reasons
- test this by going into the dir and running
mix compile
manually- if the version was right, it will have nothing to do
- a second run should take far less time than before but produce the same errors and warnings
- please report timings for both a first and second run
- you can
rm -rf repos/
to remove the repos to get "first" run timing time mix dev.setup
- you can
- downloads and builds each project correctly
- test
mix dev.update.gitlab
- it should generate a valid yml on your machine
- use this to validate it: https://gitlab.tetrarch.co/commune_bot/development/-/ci/lint
- it should have one
lint_bash
job, onedocs
job, and the following jobs repeated for each valid combination or erlang, elixir, and alpine:deps
build
credo
format
dialyzer
test
- right now, the only invalid combination of elixir and erlang in our matrix is elixir 1.12.3 and erlang 25.2.3
- its possible you will see versions newer than this, but you should not see older versions
- some combos of elixir/erlang will show up more than once since we support multiple alpine versions
- it should generate a valid yml on your machine
Parts of this MR in need of review:
- is the ux good?
- do the planned tasks cover the things you see yourself needing to do?
- is it easy to understand what each step does?
- would it be easier to understand if the logic to skip steps was moved into
Step.do_steps/3
?
- would it be easier to understand if the logic to skip steps was moved into
- is there value in providing a struct to make the structure explicit?
- is it easy to understand how the setup mix task works?
- does the idiom of composing steps into tasks make sense? is there a better one?
- is it easy to understand how the update gitlab mix task works?
Work still to do:
-
think about how to handle repos which have work in progress -
on a branch other than main -
modified files -
local commits -
unfetched remote commits -
in the middle of a rebase -
other states?
-
-
add moduledocs to each step -
clone -
compile -
credo -
deps -
dialyzer -
docs -
ready -
secrets -
tasks -
tests
-
-
add moduledocs to other modules -
development/gitlab -
helpers -
step
-
-
add longdocs to tasks -
dev.setup -
dev.update.gitlab -
consider removing -h
/--help
flag in favor of longdocs
-
-
add files and templates for planned tasks -
.tool-versions.eex -
.gitignore.eex -
mix.exs.eex -
README.md.eex -
.credo.exs -
.formatter.exs -
config/ -
config.exs -
config_with_nostrum.exs.eex
-
-
lib/ -
module.ex.eex -
bot.ex.eex -
library.ex.eex -
application.ex.eex
-
-
test/ -
test_helper.exs -
test.exs.eex
-
-
-
add git hooks -
pre-commit.sh (delegates to pre-commit.d/*) -
00-localdeps.sh - reject commit if
.localdeps
exists and mix.exs or mix.lock is staged
- reject commit if
-
credo.sh - reject commit if credo fails
-
dialyzer.sh - reject commit if dialyzer fails
-
format.sh - could either dumbly reject unformatted changes, or reformat them on the fly
- ref: https://prettier.io/docs/en/precommit.html#option-6-shell-script
- reformatting is risky since it may incorrectly handle partially staged files
- dumbly rejecting also doesnt really work on partially staged files since it rejects staged chunks if an unstaged one is unformatted
- need to do more research here
-
-
-
start planning work on other mix tasks -
open an issue for each -
goal of task -
steps to do task, in order -
planned cli -
list of concerns/considerations
-
-
Re: other issues:
- fixes #3 (moved)
- the unlinking script is not needed with the .gitlab-ci.yml generated by the new mix task
- obsoletes #1
- #1 should be closed and any tasks which are not covered by this MR should be opened as new issuesz
Anything not completed in this MR will be split out into other issues
Edited by skwerl