I’m evaluating Git X Modules with a GitHub setup. In our scenario we would potentially have repositories with many (a couple hundred) modules in it. We already have tooling to determine which versions of which modules we’d like to add to a repository. It is not very feasible to manage these using the UI.
Is there an API available to manage the modules? Adding, removing, changing the branch, etc.
Thanks
Joris
Hello Joris,
at the moment we don’t have an API to manage the modules but we plan to add it in the future. The API will have a form of a special .xmodules text file in the repository: you will be able to change it (e.g. add module description) and push the change to the repository and our server will apply these changes.
We believe that our users should own their data and be able to manage them programmatically, so we already keep all the data in the repository (in a special references). So once you backup the repository, you have all the data about the modules with you.
Sounds good! Do you have a timeline for when this feature would be available?
Hello Joris,
I think, we will add it at the beginning of the next year.
For now I can propose the following temporary work-around. Try it on a test repository first. Also note that X-Modules GitHub app should be installed (big green “+ Add Repositories” button) both parent and module repositories.
Step 1. Add some module into your future repository to make it initialized (so that the “Sync” button becomes enabled on the parent repository page). You can skip this step if you already have “Sync” button enabled. It becomes enabled once you’ve ever added a module into such a repository. It doesn’t matter which module you add, we will overwrite it in the future.
Step 2. Edit the following script to adjust your needs:
#!/bin/sh
# path to the helper Git repository if it doesn't exist
GIT_REPO=/tmp/gitdir
# branch in the parent repository, don't forget "refs/heads/" prefix; the branch must exist
PARENT_REPO_BRANCH=refs/heads/main
cd $GIT_REPO
# create the helper Git repository if it doesn't exist
if [ ! -d $GIT_REPO ] ; then
git init $GIT_REPO
fi
GIT_REPO_FILE=$GIT_REPO/file
rm -f $GIT_REPO_FILE
for i in `seq 3` ; do
# user-specified options
# obtain GitHub repository ID, e.g. using GitHub API:
GIT_REPO_ID=
if [ -z $GIT_REPO_ID ] ; then
echo "GIT_REPO_ID is required"
exit 1
fi
# path in the parent repository
MODULE_PATH=module$i
# path in the module repository, usually empty if you want to insert module repository fully, not its subtree
SUBTREE_PATH=
# branch in the module repository, don't forget "refs/heads/" prefix; the branch should exist
MODULE_REPO_BRANCH=refs/heads/main
# create pull requests in case of conflicts
CONFLICT_POLICY=request-pull
# squash option, valid values: once, always, never
SQUASH_POLICY=never
# merge option, valid values: subtree (create merge commits merging module commits), rebase (apply module commits on the top of parent commits)
MERGE_POLICY=subtree
# exclude patterns, multiline value, usually empty if there're no files to exclude (e.g. *.iso)
EXCLUDE_PATH=
# constants
GX_NAMESPACE=gx
GITHUB_NAMESPACE=gh
# derived variables
SECTION_NAME=`uuidgen`
XMODULES_URL="agnostic://$GITHUB_NAMESPACE/$GIT_REPO_ID"
BRANCH_RULE="if $PARENT_REPO_BRANCH sync $MODULE_REPO_BRANCH"
SQUASH_RULE="squash $SQUASH_POLICY"
MERGE_RULE="merge $MERGE_POLICY"
REQUEST_REF="refs/$GX_NAMESPACE/$PARENT_REPO_BRANCH"
# construct the config file
git config --file $GIT_REPO_FILE module.$SECTION_NAME.path "$MODULE_PATH"
git config --file $GIT_REPO_FILE module.$SECTION_NAME.url "$XMODULES_URL"
git config --file $GIT_REPO_FILE module.$SECTION_NAME.ref "$BRANCH_RULE"
git config --file $GIT_REPO_FILE module.$SECTION_NAME.conflict $CONFLICT_POLICY
git config --file $GIT_REPO_FILE module.$SECTION_NAME.pull "$SQUASH_RULE"
git config --file $GIT_REPO_FILE --add module.$SECTION_NAME.pull "$MERGE_RULE"
if [ ! -z "$SUBTREE_PATH" ] ; then
git config --file $GIT_REPO_FILE module.$SECTION_NAME.subtree "$SUBTREE_PATH"
fi
if [ ! -z "$EXCLUDE_PATH" ] ; then
git config --file $GIT_REPO_FILE module.$SECTION_NAME.excludePath "$EXCLUDE_PATH"
fi
done
# write the config file into the object database as a blob
BLOB_ID=`git hash-object -w -t blob $GIT_REPO_FILE`
# set request ref on it, e.g. if the parent branch is refs/heads/master, the request ref is refs/gx/refs/heads/master (yes, "refs/" are mentioned twice)
git update-ref $REQUEST_REF $BLOB_ID
echo "========================================================================="
echo "Now 'cd' into $GIT_REPO and run"
echo ""
echo " git push <YOUR_PARENT_REPO_URL> $REQUEST_REF"
echo ""
echo "========================================================================="
This example script adds 3 modules pointing to the same branch of the same module repository (specified in GIT_REPO_ID variable) into “module$i” path i.e. module1, module2, and module3 subdirectories. In your real case for every iteration of the cycle the module repository ID (GIT_REPO_ID, a 9-digits number) will be different as well as MODULE_PATH and MODULE_REPO_BRANCH.
SUBTREE_PATH corresponds to “Root path” of the UI.
SQUASH_POLICY and MERGE_POLICY correspond to “Squash commits” and “Merge Strategy” options of the UI.
MODULE_PATH corresponds to “Repository Path” of the UI.
GIT_REPO_ID and MODULE_REPO_BRANCH define “Module Location” options.
The branch in the parent repository is defined in PARENT_REPO_BRANCH. The script doesn’t contain information about the parent repository at all. It will generate a special reference (e.g. refs/gx/refs/heads/main, the name depends on the PARENT_REPO_BRANCH) in the temporary Git repository at GIT_REPO. You will then push that reference from that temporary Git repository to the parent repository.
Step 3. Run the adjusted script
sh add-xmodules.sh
It will print:
=========================================================================
Now 'cd' into /tmp/gitdir and run
git push <YOUR_PARENT_REPO_URL> refs/gx/refs/heads/main
=========================================================================
Step 4. cd into /tmp/gitdir
There will be a file named “file”. It would look like:
[module "66ed8c0e-ac56-4d29-a570-0a5dfb4335d0"]
path = module1
url = agnostic://gh/427389897
ref = if refs/heads/main sync refs/heads/main
conflict = request-pull
pull = squash never
pull = merge subtree
[module "16b2e346-a52c-48f6-bdf7-52b85c38f283"]
path = module2
url = agnostic://gh/427389897
ref = if refs/heads/main sync refs/heads/main
conflict = request-pull
pull = squash never
pull = merge subtree
[module "21c02a12-90b3-49b4-93a1-333f7875cad7"]
path = module3
url = agnostic://gh/427389897
ref = if refs/heads/main sync refs/heads/main
conflict = request-pull
pull = squash never
pull = merge subtree
427389897 — is my module repository id, I’ve inserted it 3 times.
Check that everything looks ok.
Step 5. Push the reference into the parent repository as the script recommends. The name of the reference will depend on PARENT_REPO_BRANCH.
Step 6. Click “Sync” button on the parent repository page. When the sync is complete, refresh the page. The original module(s) should disappear, they should be replaced with the modules specified in the script.
When we implement a special file feature, the file will look similar, but it will reside in the working tree directly.
When you click “Sync” button, our engine interprets refs/gx/refs/heads/master (or any other refs/heads/<FULL_REF_NAME>) reference pointing to a Git blob with the config above as a command to replace all the modules with the modules this blob describes. Once gitx.dev runs this command, it deletes that refs/gx/… reference.
How to get module repository ID (GIT_REPO_ID)? It’s GitHub repository ID, it’s a 9-digits number.
You can find it on the repository page of gitx.dev UI. Or you can also get it with GitHub REST API by name, e.g. the following GitHub REST API command lists of all the user’s repositories (the user it identified by GitHub token, usually starting with “ghp_…”)
curl -H "Authorization: token <GITHUB_TOKEN>" -H "Accept: application/vnd.github.v3+json" https://api.github.com/user/repos
or you can look it up by “username/reponame” pair using the following request:
curl -H "Authorization: token <GITHUB_TOKEN>" -H "Accept: application/vnd.github.v3+json" https://api.github.com/repos/username/reponame
If everything above it too tricky for you, you can wait for the “special .xmodules file” API, but it will be very similar to the approach above (the content of the .xmodules file will be similar to the content above).
Thank you for the detailed answer, I will look into this.