Ticket #1035 (closed enhancement: fixed)

Opened 8 years ago

Last modified 7 years ago

Switch source code to Git

Reported by: dscho Owned by: dscho
Priority: minor Milestone: imagej2-b3-headless
Component: Server Admin Version:
Severity: minor Keywords:
Cc: ctrueden@… Blocked By:
Blocking:

Description

Once upon a time, we already had it, and it makes collaborative development much easier for us.

The problem with it is that Trac-Git seems to be slow. We need to look into how to speed up things there.

Change History

comment:1 Changed 8 years ago by dscho

  • Owner changed from curtis to dscho
  • Status changed from new to accepted

comment:2 Changed 8 years ago by dscho

  • Cc ctrueden@… added

Curtis reports that the performance improved with switching the Git plugin's caching *off*.

The major problem now seems that we need a way to somehow deal with the many references to Subversion revisions. Either we convert them to hashes (which would be a one-time, clean break) or we hack something like "r<n> -> 79599fe5f~<4992 minus n>".

comment:3 Changed 8 years ago by curtis

My vote is for the one-time conversion. Trac now allows editing of both description and individual comments, so it should be doable to write a script that updates all "r\d+" patterns to their corresponding Git hash. Then we don't have to hack the commit hashes themselves, and the Trac links will not be broken.

comment:4 Changed 8 years ago by dscho

One thing we probably want to do is to activate the Trac plugin on  https://github.com/imagej/imagej/admin/hooks to update the ticket statuses automatically.

comment:5 Changed 7 years ago by dscho

  • Status changed from accepted to closed
  • Resolution set to fixed

We used the script bin/rewrite-svn-branch.perl to rewrite all of our branches and replaced 'master' with that history in in 2defeef21c2c2dc5c96a79832e01adc94d8466ce. The conversion script was subsequently removed (but it is still part of history).

To rewrite the tickets in Trac, we used this script:

#!/bin/sh

PERL=/tmp/trac-svn2git.$$.perl
TRAC=/var/lib/trac/imagej/db/trac.db

cat > $PERL << EOF
our %mapping = (
EOF

(cd $HOME/ij2 &&
        git log master |
        perl -e '
                my $commit = "";
                my $sep = "";
                while (<>) {
                        if (/^commit ([0-9a-f]{40})/) {
                                $commit = $1;
                        }
                        elsif (/^    This used to be revision (r[1-9][0-9]*)\.$/) {
                                print $sep . "\"" . $1 . "\" => \"" . $commit . "\"";
                                $sep = ",\n";
                        }
                }
                print "\n";
        ') >> $PERL

cat >> $PERL << \EOF
);

sub map_revision ($) {
        my $text = $_[0];
        my @list = split(/\b(r[1-9][0-9]*)\b/s, $text);
        for (my $i = 1; $i <= $#list; $i += 2) {
                if (exists($mapping{$list[$i]})) {
                        $list[$i] = $mapping{$list[$i]};
                }
        }
        return join('', @list);
}

while (<>) {
        while ((tr/'// % 2) == 1) {
                $_ .= <>;
        }
        if (/^INSERT INTO "ticket" VALUES\((\d+),(NULL|'([^']*)'),(\d+),(\d+),(NULL|'([^']*)'),(NULL|'([^']*)'),(NULL|'([^']*)'),(NULL|'([^']*)'),(NULL|'([^']*)'),(NULL|'([^']*)'),(NULL|'([^']*)'),(NULL|'([^']*)'),(NULL|'([^']*)'),(NULL|'([^']*)'),(NULL|'(([^']|'')*)'),(NULL|'(([^']|'')*)'),(NULL|'([^']*)')\);$/s) {
                my $id = $1;
                my $type = $2;
                my $time = $4;
                my $changetime = $5;
                my $component = $6;
                my $severity = $8;
                my $priority = $10;
                my $owner = $12;
                my $reporter = $14;
                my $cc = $16;
                my $version = $18;
                my $milestone = $20;
                my $status = $22;
                my $resolution = $24;
                my $summary = $26;
                my $description = $29;
                my $keywords = $33;

                my $new_summary = map_revision($summary);
                if ($new_summary ne $summary) {
                        print "UPDATE ticket SET summary=" . $new_summary . " WHERE id=" . $id . ";\n";
                }
                my $new_description = map_revision($description);
                if ($new_description ne $description) {
                        print "UPDATE ticket SET description=" . $new_description . " WHERE id=" . $id . ";\n";
                }
        }
        elsif (/^INSERT INTO "ticket_change" VALUES\((\d+),(\d+),(NULL|'([^']*)'),(NULL|'([^']*)'),(NULL|'(([^']|'')*)'),(NULL|'(([^']|'')*)')\);$/s) {
                my $ticket = $1;
                my $time = $2;
                my $author = $3;
                my $field = $5;
                my $oldvalue = $7;
                my $newvalue = $10;

                my $new_oldvalue = map_revision($oldvalue);
                if ($new_oldvalue ne $oldvalue) {
                        print "UPDATE ticket_change SET oldvalue=" . $new_oldvalue . " WHERE ticket=" . $ticket . " AND time=" . $time . " AND field=" . $field . ";\n";
                }
                my $new_newvalue = map_revision($newvalue);
                if ($new_newvalue ne $newvalue) {
                        print "UPDATE ticket_change SET newvalue=" . $new_newvalue . " WHERE ticket=" . $ticket . " AND time=" . $time . " AND field=" . $field . ";\n";
                }
        }
}
EOF

commands="$(printf  '.dump ticket\n.dump ticket_change\n' |
sqlite3 "$TRAC" |
perl $PERL)"

echo "$commands" |
sqlite3 "$TRAC"

In short, this script looks for the mappings between Subversion and Git revisions (during the rewrite, the git-svn id line would be replaced by something parse-able like "This used to be revision ....") and then scans the summary and description fields in the ticket table and the oldvalue and newvalue fields in the ticket_change table for rewritten Subversion tickets (it leaves those alone for which we do not have Git revisions; this happened e.g. for loci-java commits), writes appropriate UPDATE statements and finally feeds those to SQLite3.

comment:6 Changed 7 years ago by curtis

  • Milestone changed from imagej-2.0.0 to imagej-2.0.0-beta3
Note: See TracTickets for help on using tickets.