Automated tests

Git X-Modules
#1

Hi,

I was wondering if it would be possible to run some automated tests on an X-Module after a push, before updating its external repository. Thanks!

#2

Hi Panolo,

Thanks for the great question! Could you elaborate more on your current setup:

  • What X-Modules tool do you use? Bitbucket plugin or the command-line tool?
  • How do you run automated tests for repositories that don’t have X-Modules? Do you rely on pre-receive hooks, pull requests, or anything else?

At the moment the command-line tool is lagging behind the X-Modules plugin for Bitbucket. So, my further comments mostly apply to the Bitbucket plugin.

We’re considering the following options for your particular scenario:

  1. Pre-receive hooks:

    When you push a change to a parent repository, X-Modules triggers a push to an external repository as well. That means you can run the automated tests at a pre-receive hook installed at the external repository. The push to the external repository will proceed only when all the necessary tests pass.

  2. Temporary branches:

    Both parent and external repositories may have a couple of branches, e.g. pre-tested and master, and X-Modules must be configured for both these branches, so pre-tested in parent is synced with pre-tested in the external repository, master is synced to master.

    Before publishing any changes to master of the parent repository, you can push the changes to pre-tested branch first. As result, the external repository receives corresponding updates at its own pre-tested branch and so you can run the automated tests on this branch; if all tests are good, then you can merge pre-tested into master at parent repository and that will update master branch in the external repository as well.

  3. Module pull-requests:

    This is something we’re working on: you’ll be able to publish changes to an external repository as a pull request rather than a regular branch update. That means you can push a change to your parent repository and then our tool will extract the changes corresponding to the X-Module and create a pull-request with these changes at the external repository. As result, you’ll be able to run the automated tests on a pull request and then merge it if the tests pass.

    Unfortunatley, this feature is unavailable at the moment, but we have a proof-of-concept for it.

  4. Dry-run push:

    We’ve been discussing the following option within our team as well: you could push an update to the parent repository with a dry-run push option. As result, X-Modules will extract the changes and create special dry-run refs with corresponding commits in the parent repository without updating the refs you were pushing to. After that you can run the tests on those dry-run refs.

    This option is far from being implemented, we’re just considering that.

Let us know what do you think about these options. What option’d feet your workflow better? Your feedback is very welcome.

#3

Thanks for the detailed reply, it was really helpful!

I’m currently testing out the CLI, waiting for the Gitlab plugin to be released.

Automated tests are something that I’d like to start using for my future projects, as part of a new CI/CD pipeline, so nothing is yet in place.

That said, options 1 and 3 are probably what I was looking for. Is the Gitlab plugin going to support these features? Can you give me a rough ETA? Thanks again!

#4

Thanks for the update!

We’re putting a lot of effort into GitLab integration at the moment, both cloud and self-hosted options. We’re going to make an alpha/beta version of the app available later this summer and it’ll take several more months before the stable version arrives.

That said, options 1 and 3 are probably what I was looking for. Is the Gitlab plugin going to support these features?

Option 1 will be available right away and we already support GitLab merge requests for resolving possible conflicts in parent and external repositories, however, full-featured option 3 will take some more time, probably, 1-2 months after the initial alpha/beta release.

I have a couple more questions if you don’t mind:

  1. Are going to use GitLab Cloud or GitLab On-premises?

  2. We’re going to publish GitLab integration as a couple of solutions:

    • web service that you can connect with your GitLab Cloud repositories;

    • docker image that you can deploy against GitLab Cloud or self-hosted GitLab instance.

    What solution would work for you better?

#5

I’m using a self-hosted GitLab instance. Does the web service work for my case? Either way, both solutions seem good.

#6

Does the web service work for my case?

Not at the moment, but we’ll keep this case in mind.

We’ll keep you updated on our progress. You can also follow us on Twitter @gitmodules, we post announcements there.

#7

Hi,

I’m now considering another scenario in which all new changes are periodically collected and tested in a single batch, instead of running a test after every single push.

Option 3 would be great if it could get all new commits and create a single pull request. Do you think that would be feasible?

Of course alternative solutions are more than welcome. Thanks!

#8

Option 3 would be great if it could get all new commits and create a single pull request. Do you think that would be feasible?

So far, we’re going to support per-module pull requests, i.e. one pull request for every module. We didn’t think about a batched pull request that has updates from various modules. I believe it’s totally doable, yet we have to think it over and discuss it within our team.

I wonder if it’s possible to emulate such pull requests through batching pull requests through GitLab, so X-Modules create per-module pull requests and then you join them somehow with GitLab means. Do you know any GitLab feature relevant to that?

#9

Sorry, I actually mean grouping new commits in a module in a single pull request, instead of creating a pull request for every commit. It would be great if it was already intended to do that.

#10

Ah, right. This is the way we’ve implemented that in the prototype.

#11

By the way, if I start using the CLI on a project, will the migration to the plugin be easy?

#12

We try to make the migration straightforward. You’d need to transfer Git X-Modules metadata to the new app which is just a push to a new repository and you’d probably need to update the module URLs, that’s basically it.

Also, I want to remind you that Git X-Modules is still in its beta stage and we don’t recommend you to use it on live production repositories.