Ticket #1731 (closed defect: fixed)

Opened 6 years ago

Last modified 6 years ago

Make sure Fiji infrastructure is properly backed up

Reported by: dscho Owned by: dscho
Priority: major Milestone: imagej2-b7-ndim-data
Component: Server Admin Version:
Severity: serious Keywords:
Cc: curtis Blocked By:
Blocking: #1729, #1757

Description

Probably as a Jenkins job, we should make sure that fiji.sc things are written properly to the /backup/ directory. The following databases and file system dumps need to be made:

  • Fiji Wiki (including images/, extensions/ and skins/)
  • Bugzilla
  • IRC logs (shinybot)
  • Git mirrors (or remove them?)

Change History

comment:1 Changed 6 years ago by dscho

Bugzilla needs the bugzilla MySQL database, the /var/www/bugzilla/.git repository and the /etc/apache2/conf.d/bugzilla file. The libapache2-mod-perl2 module needs to be enabled, too.

In addition, when re-installing, the lib/ directory needs to be blown away and install-modules.pl --all needs to be run as root.

Last edited 6 years ago by dscho (previous) (diff)

comment:2 Changed 6 years ago by dscho

Fiji Wiki requires the wikidb3 database, the /var/www/mediawiki/phase3/images/ directory (owned by www-data.www-data) and the /var/www/mediawiki/{,phase3/extensions/,phase3/skins/}.git repositories.

comment:3 Changed 6 years ago by dscho

  • Blocking 1757 added

(In #1757) It seems that in addition to #1733, this required copying fiji-dot.png and openlayers/ to the appropriate directory.

Leaving this open until #1731 is fixed (and includes backing up the stuff for the usage map).

comment:4 Changed 6 years ago by dscho

To be precise, the usage map requires the file fiji-dot.png and the Git checkout openlayers/ in awesome/ to be backed up, too.

While at it, favicon.ico and downloads/ should be backed up for the Fiji Wiki to work, too. Oh, and LocalSettings.php would be rather crucial, too...

comment:5 Changed 6 years ago by dscho

For GitWeb to work, the static symlink and httpd.include are important. The repositories are expected to live in /data/git, and Git is expected to be installed from Git for Windows' source ( https://github.com/msysgit/git); The httpd.include includes all the information required to run the smart HTTP backend and GitWeb.

comment:6 Changed 6 years ago by dscho

  • Milestone changed from imagej2-unscheduled to imagej2-b7-ndim-data

comment:7 Changed 6 years ago by dscho

The code search facility (e.g.  http://fiji.sc/IJ.java) requires cgi-bin/search.cgi to be backed up in addition to httpd.include.

comment:8 Changed 6 years ago by dscho

Fiji's samples reside in the samples/ directory which also needs to be backed up.

comment:10 Changed 6 years ago by dscho

The uploaded sample images from  http://upload.imagej.net/ need to be backed up, too (since Fiji's Help>Upload Sample Image uses ImageJ2's command to do the job).

comment:11 Changed 6 years ago by dscho

The Backup-fiji.sc job now makes sure that the Wiki database, the BugZilla database, the Geolocation database, the Wiki files, the virtual host configuration, and the GitWeb configuration are written into timestamped files in /backup/fiji/.

It also makes sure that files that are not supposed to change are rsync'ed into /backup/fiji-eden/' (to avoid unnecessary copying). These files are in downloads/, images/, javadoc/, samples/ and update/.

Finally, it clones the .git/ directories into bare mirror repositories in /backup/fiji-eden/.

Once all that is done, the Backup-fiji.sc job cleans up files older than 2 weeks in /backup/fiji.

The Backup-Uploaded-Sample-Images job backs those images up, into /backup/sample-images/. I also copied the uploaded sample images from pacific.

It appears that rsync is really slow, despite using the -u option (only newer files)... :-(

comment:12 Changed 6 years ago by dscho

  • Status changed from new to closed
  • Resolution set to fixed

Luckily, rsync is only slow the first time. The Backup-fiji.sc job took 1h11min for the run that finally put all things into the correct places (actually, the Git repositories were already cloned by the same script but outside Jenkins), but now it only takes 2min45sec.

At the time of writing, the Backup-Uploaded-Sample-Images job still is crunching away on its initial run.

Note: See TracTickets for help on using tickets.