Ticket #1863 (closed defect: fixed)

Opened 6 years ago

Last modified 6 years ago

Fix Jenkins' increasingly frequent "out of PermSpace" errors

Reported by: dscho Owned by: dscho
Priority: critical Milestone: imagej2-b7-ndim-data
Component: Server Admin Version:
Severity: critical Keywords:
Cc: curtis Blocked By:
Blocking: #1742

Description

It seems that for some days (weeks?) Jenkins freezes at some stage, running out of PermSpace. We need to fix this.

Change History

comment:1 Changed 6 years ago by dscho

The reason seems to be the hack in Stable-Fiji to generate pseudo SVn changes; to do so, there is a Groovy Postbuild script to that instantiates URLClassLoaders pretty frequently...

Last edited 6 years ago by curtis (previous) (diff)

comment:2 Changed 6 years ago by dscho

In the process of working on this issue, I wrote a little script to virtually click Build Now on all currently-failed jobs:

#!/usr/bin/jenkins-cli groovy

map = jenkins.model.Jenkins.instance.getItemMap()
map.each() {
        name, item ->
                if (item instanceof hudson.model.Job) {
                        if (item.getLastBuild().getResult().isWorseOrEqualTo(hudson.model.Result.FAILURE)) {
                                item.scheduleBuild(new hudson.model.Cause.RemoteCause("localhost", "Started by rebuild-all-failed.groovy"))
                        }
                }
}

It requires the JENKINS_URL environment variable to be set; adding '-s <URL>' to the shebang line fails for some obscure reason (jenkins-cli claims that it does not know the command).

comment:3 Changed 6 years ago by dscho

  • Status changed from new to closed
  • Resolution set to fixed

Inspired by  https://issues.jenkins-ci.org/secure/attachment/20266/GroovyCommand_patch.txt (attached to  https://issues.jenkins-ci.org/browse/JENKINS-8892) I now use Jenkins' own "uber" class loader:

 http://jenkins.imagej.net/job/Stable-Fiji/jobConfigHistory/showDiffFiles?histDir1=%2Fvar%2Flib%2Fjenkins%2Fattic%2Fjobs%2FStable-Fiji%2F2013-05-10_14-39-53&histDir2=%2Fvar%2Flib%2Fjenkins%2Fattic%2Fjobs%2FStable-Fiji%2F2013-05-11_11-26-05

This should help with PermSpace being cluttered.

In case this is not enough, we need to add the Java options -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled to JAVA_ARGS in /etc/default/jenkins.

But let's do that only if we absolutely must, since I have no idea how that interferes (or not) with webapp-style class loading. So I'll close this for now, to be reopened if the problem actually persists.

Note: See TracTickets for help on using tickets.