Ticket #1783 (closed feature: fixed)

Opened 6 years ago

Last modified 5 years ago

Make version binding tighter for reproducible builds

Reported by: dscho Owned by: dscho
Priority: major Milestone: imagej-2.5.0
Component: Build System Version:
Severity: major Keywords:
Cc: curtis Blocked By: #1784, #1785, #1786, #1787, #1788
Blocking:

Description

ImageJ2 is not a stand-alone project; it depends on ImgLib2, scijava-common, pom-scijava and SCIFIO. Fiji depends on all of them.

Builds, however, are not reproducible (as can be seen by failing git bisect and Jenkins' Fiji build failures). The reason is that there are -SNAPSHOT version couplings.

Goal: get rid of all -SNAPSHOT dependencies in all of above-mentioned projects.

To maintain ease of development, this implies introducing some tools:

  • just like  http://jenkins.imagej.net/job/Synchronize-and-deploy-IJ1/ synchronizes ImageJ 1.x fully automatically, commits, pushes and deploys, we need Jenkins jobs (most likely manually triggered, however) to help bumping up the version in one of our projects.
  • the Jenkins job should also trigger dependent projects to update, e.g. a scijava-common bump should bump pom-scijava which in turn should bump the rest of 'em.
  • this Jenkins job should avoid pushing updated 'master' branches when builds fail, but simply mark the builds as failed.
  • we need a Jenkins job to warn about API changes in -SNAPSHOT versions. Most likely, this job should make use of the Animal Sniffer Maven plugin:  http://mojo.codehaus.org/animal-sniffer-maven-plugin/.
  • the Animal Sniffer job could automatically push branches as appropriate for micro, minor and major version increases to a special namespace (e.g. refs/suggested/) to ease the maintenance burden.
  • it would probably be a good idea to extend  http://jenkins.imagej.net/job/ImageJ-style/ to check also for SNAPSHOT dependencies pushed to master by mistake.
  •  http://jenkins.imagej.net/job/ImageJ-style/ could be extended to fix easy-to-fix problems and commit & push them to master.

The idea is really to help the development process without adding a substantial burden on the developers. Tools should be helpful, not create more work.

Change History

comment:1 Changed 6 years ago by dscho

Ah, and of course: the Jenkins jobs should be written as .sh scripts in scijava-common's bin/ directory for transparency.

comment:2 Changed 6 years ago by curtis

We will also write up a public rationale for how and why we do versioning this way for SciJava projects (SciJava Common, SCIFIO, ImgLib, ImageJ, Fiji,  LOCI projects, etc.):

  • Conformity to  SemVer versioning system.
  • Fixed release version couplings (i.e., no SNAPSHOTs, no version ranges) for build reproducibility, and hence bisectability, along with many other benefits.
  • There is nothing wrong with SNAPSHOT or version range couplings, as long as you understand the tradeoffs. Any downstream projects can continue to use these with SciJava projects, as long as they understand that their builds will not be reproducible.
  • It is not our intent to enforce any particular workflow on people developing on master. If people want to break API left and right during development on master, that is their prerogative. However, it does incur a slight (temporary) technical debt when making the next release, since either: A) any breaking changes would need to be "undone" (i.e., readding any previous method signatures with deprecation tags); or B) the next release would need a major version bump indicating lack of backwards compatibility. Obviously (A) is preferred, and the Animal Sniffer Maven plugin and related tooling (discussed above) will be very helpful in assisting in pursuit of (A).
  • Of course, it is strongly preferred that for each commit to master (including all commits as part of a merge commit), code still compile, hence the CI jobs for testing this. For a variety of reasons, broken builds on master should be fixed as soon as possible.
  • The only new requirement process-wise is that code on master should not introduce any SNAPSHOT dependencies, because those commits will then not have a reproducible build. Any SNAPSHOT dependencies on master should be reverted to fixed release dependencies as soon as possible, just like failing builds. We may create a Jenkins job to check this, which fails in the case of invalid dependency versions. (To put another way: we want it to be possible to cut a release from master as easily as possible. This means code with only fixed release dependencies, which compiles and has passing unit tests.)

Lastly, the  versions-maven-plugin is a great tool for verifying that you are using the latest release version of both Maven plugins and project dependencies. We can make use of it when performing prerelease checks.

Last edited 6 years ago by curtis (previous) (diff)

comment:3 Changed 6 years ago by curtis

One other thing that dscho & I discussed: we ideally want to use the  Maven Release plugin to perform our releases. However, the default behavior of mvn release:prepare cuts the release on master and then pushes it. We would prefer to cut the release to the (orphaned) tag itself, and only commit the SNAPSHOT version bump on master. I.e., no commit on master should have a release version in the POM. This is how we've been doing it with our  tag-release.sh script. The mvn release:branch goal might be a solution there; further investigation should be a subticket of this feature.

comment:4 Changed 6 years ago by dscho

  • Blocked By 1784 added

comment:5 Changed 6 years ago by dscho

  • Blocked By 1785 added

comment:6 Changed 6 years ago by dscho

  • Blocked By 1786 added

comment:7 Changed 6 years ago by dscho

  • Blocked By 1787 added

comment:8 Changed 6 years ago by dscho

  • Blocked By 1788 added

comment:9 Changed 5 years ago by curtis

  • Status changed from new to closed
  • Resolution set to fixed

We did it! SciJava, SCIFIO, ImgLib2 and ImageJ now use only release couplings between components.

There is still some work to be done splitting ImgLib2 to separate repositories, as well as getting Fiji fully transitioned away from snapshot couplings, but that will come with time. No need to keep this ticket around reminding us.

Note: See TracTickets for help on using tickets.