Overcoming Fear: 2011

Thursday, November 10, 2011

Trampoline and Memoize

Goal

Groovy 1.8 introduces two new closure functions. Memoization is the automatic caching of closure results. Trampolining permits a form of declarative tail-call optimization. This article introduces the two concepts and demonstrates how to combine them in order to create cached, tail-recursive closures.

tl;dr

The example code from this article is available on github.

Simple Memoization

Creating a closure that caches the result of some calculation is as easy as appending .memoize() or one of the alternate memoize...(...) methods that allows more fine-grained control over cache sizes to a closure declaration. One benefit of memoization includes the caching of results of long-running calculations that have no side effects. Memory leaks are a potential pitfall, which is why a maximum cache size should generally be prefered.
The specification below contains a closure with a side-effect. This side effect happens just once, despite the closure being invoked twice.

SimpleMemoizationSpec.groovy

package timezra.groovy.trampoline_memoize

class SimpleMemoizationSpec extends spock.lang.Specification {

  int count

  def identity = {
    count++
    it
  }.memoize()

  def "each call should be cached"() {
    when:
    def first = identity 0
    def second = identity 0

    then:
    count == 1
    first == second
    second == 0
  }
}

Recursive Memoization

Suppose we want to memoize the results of a recursive closure call. For example, we can unroll the call tree of this naive implementation of the fibonacci function.

  def fib = { n ->
    if(n == 0) 0
    else if(n == 1) 1
    else fib(n-1) + fib(n-2)
  }

The call trace for the fourth fibonacci number looks like this.

                       ___________fib 4___________
                      /                           \
               fib 3 + fib 2                 fib 2 + fib 1
             /               \                 /
      fib 2 + fib 1     fib 2 + fib 1     fib 1 + fib 0
       /
fib 1 + fib 0

NB: The closure here is entered nine times but we can see that it only needs to be entered 5 times because 4 calls are repeated. If we cache the results of the fibonacci calls, the complexity of even a naive implementation such as this will increase linearly, rather than exponentially.

Unfortunately, because, in Groovy 1.8, a closure cannot invoke another closure directly, memoizing this closure is not entirely straightforward. The example directly below does not work.

RecursiveMemoizationSpec.groovy

package timezra.groovy.trampoline_memoize

class RecursiveMemoizationSpec extends spock.lang.Specification {

  int count

  def fib = { n ->
    count++
    if(n == 0) 0
    else if(n == 1) 1
    else fib(n-1) + fib(n-2)
  }.memoize()

  def "calls should be cached"() {
    when:
    def actual = fib 10

    then:
    actual == 55
    count == 11
  }
}

The stack trace when a closure calls itself.

groovy.lang.MissingMethodException: No signature of method: org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.doCall() is applicable for argument types: (java.lang.Integer) values: [9]
Possible solutions: call(), call([Ljava.lang.Object;), call(java.lang.Object), call([Ljava.lang.Object;), findAll(), equals(java.lang.Object)
  at timezra.groovy.trampoline_memoize.RecursiveMemoizationSpec.$spock_initializeFields_closure1(RecursiveMemoizationSpec.groovy:11)
  at groovy.lang.Closure.call(Closure.java:410)
  at groovy.lang.Closure.call(Closure.java:423)
  at timezra.groovy.trampoline_memoize.RecursiveMemoizationSpec.calls should be cached(RecursiveMemoizationSpec.groovy:16)

Since methods can invoke memoized closures, the solution is to invoke the call method on the closure.

RecursiveMemoizationSpec.groovy

class RecursiveMemoizationSpec extends spock.lang.Specification {
....
  def fib = { n ->
    count++
    if(n == 0) 0
    else if(n == 1) 1
    else fib.call(n-1) + fib.call(n-2)
  }.memoize()
....
}

NB: The un-memoized version enters the closure 177 times, but the memoized version enters just 11.

Trampoline

Declarative tail-call optimization is as simple as adding .trampoline() to a closure declaration and ensuring that recursive calls to the closure invoke the trampoline method on the closure instead of the closure itself. Some problems are more clearly solved with recursive solutions, but without automatic tail-call optimization in the JVM, recursion can quickly lead to an explosion in the size of the call stack. Trampolining is one work-around for this design tradeoff (or defect).
A tail-recursive fibonacci closure:

  def fib = { n, a = 0, b = 1 ->
    if(n == 0) a
    else fib n - 1, b, a + b
  }

By tracing the call stack, we can see its linear growth without memoization.

fib 4
  |
fib 3, 1, 1
  |
fib 2, 1, 2
  |
fib 1, 2, 3
  |
fib 0, 3, 5

In order to avoid a java.lang.StackOverflowError for sufficiently large inputs, the tail-recursive closure must be explicitly trampolined.
TrampolineSpec.groovy

package timezra.groovy.trampoline_memoize

class TrampolineSpec extends spock.lang.Specification {

  int count

  def fib = { n, a = 0, b = 1 ->
    count++
    if(n == 0) a
    else fib.trampoline n - 1, b, a + b
  }.trampoline()

  def "tail calls chould be optimized"() {
    when:
    def actual = fib 1000

    then:
    actual == 1556111435
    count == 1001
  }
}

In this example, the trampolined closure is called 1001 times. If a method in Java were to call itself 1001 times, the stack would be overwhelmed. By trampolining, Groovy turns this recursion into iteration.

Memoizing a Trampolined Closure

Suppose we want to cache the results of a computationally expensive tail-recursive closure with no side effects. A trampolined closure that calls itself can easily be memoized in a separate closure declaration.

OneTimeTrampolineMemoizationSpec.groovy

package timezra.groovy.trampoline_memoize

class OneTimeTrampolineMemoizationSpec extends spock.lang.Specification {

  int count

  def fib_aux = { n, a = 0, b = 1 ->
    count++
    if(n == 0) a
    else fib_aux.trampoline n - 1, b, a + b
  }.trampoline()

  def fib = fib_aux.memoize()

  def "top-level trampolined calls should be cached"() {
    when:
    def first = fib 1000
    def second = fib 1000

    then:
    count == 1001
    first == second
    second == 1556111435
  }
}

NB: This solution caches the top-level trampolined closure, not the results of the intermediate calls.

Trampolining a Memoized Closure

Suppose we want to cache the intermediate results of a trampolined function call. For example, in our trace above, suppose we want to cache the results of fib 4 and fib 3, 1, 1 and fib 2, 1, 2 and fib 1, 2, 3 and fib 0, 3, 5.
Again, this situation is not straightforward because no closure can call a memoized closure directly, so in this case, the trampolined closure cannot call a memoized version of itself directly. Again, we can use the call function on the memoized closure.

FullTrampolineMemoizationSpec.groovy

package timezra.groovy.trampoline_memoize

class FullTrampolineMemoizationSpec extends spock.lang.Specification {

  int count

  def fib_aux = { n, a, b ->
    count++
    if(n == 0) a
    else fib.trampoline n - 1, b, a + b
  }.memoize()

  def fib = { n, a = 0, b = 1 ->
    fib_aux.call n, a, b
  }.trampoline()

  def "all trampolined calls should be cached"() {
    when:
    def first = fib 1000
    def second = fib 500, 315178285, -1898383934

    then:
    count == 1001
    first == second
    second == 1556111435
  }
}

Conclusion

Trampolining and memoization are two powerful new features in Groovy 1.8, but the combination of the two is not always straightforward. This tutorial has presented strategies for combining the two and for working around some of the limitations of their combination.

Wednesday, October 19, 2011

Tycho Archetype

Goal

The purpose of this blog post is to demonstrate how to generalize a Maven project that suits a specific need into a re-usable archetype, how to publish this archetype to a Maven repository and how to use it as the starting point for similar projects.

tl;dr

The artifact that is the end result of this tutorial can be used for creating a boilerplate Tycho plug-in project hierarchy with minimal configuration. This generated project will contain a simple Eclipse UI plug-in, an integration-test fragment, a feature that contains the binary plug-in, a feature that contains the source of the plug-in and its test fragment, and an update site for publishing both features.

Generate Archetype From Project

Since we are moving from the specific to the general, we begin by cloning the save-actions-extensions project from github.

git clone git://github.com/timezra/save-actions-extensions.git

Since we need only a few of the components from the save-actions-extensions project, we can prune out the extra. We must also rename the features, plugins and update-site directories to match the artifactIds in their respective poms along with the module declarations in the root pom. This directory structure follows the Eclipse convention, but archetype generation will be easier if we make sure that the artifactIds and directory names match. We will also rename the individual plugins and features along with their artifactIds so that there is a consistent name prefix for all projects. Again, this convention will help when generating the archetype.

The pruned project

We are now ready to generate, install and test the archetype from maven.

NB: even when specifying filtered extensions, not all files (here README, MANIFEST.MF and site.xsl) are turned into velocity templates, and the hard-coded artifactId references in some files are not all replaced with parameters.

  mvn archetype:create-from-project -Darchetype.filteredExtensions=java,html,xsl,properties,xml,MF
  cd target/generated-sources/archetype/
  find ./src -type f $ -name "*.MF" -o -name "*.xsl" $ -print0 |xargs -0 perl -pi -e 's/\$/\${symbol_dollar}/g'
  find ./src -type f $ -name "*.MF" -o -name "*.xsl" $ -exec perl -pi -e "print qq(#set( \\\$symbol_pound = '#' )\n#set( \\\$symbol_dollar = '\\$' )\n#set( \\\$symbol_escape = '\\\' )\n) if $. == 1" {} \;
  find ./src -type f -print0 |xargs -0 perl -pi -e 's/timezra\.eclipse\.save_actions_extensions/\$\{rootArtifactId}/g'
  find ./src -type f -print0 |xargs -0 perl -pi -e 's/\$\{groupId}\.save_actions_extensions/\$\{rootArtifactId}/g'
  find ./src -name archetype-metadata.xml -exec perl -pi -e 'undef $/; $_=<>; s/\s*\n\s*//g; s~README~README~g;s~META-INF~META-INF~g;s~web\*\*/\*\.xsl~web**/*.xslweb~g' {} \;
  mvn clean install
  cd /path/to/tmp/archetype
  mvn archetype:generate -DarchetypeCatalog=local
    # Choose the new archetype (here local -> timezra.eclipse:timezra.eclipse.save_actions_extensions-archetype)
    # Define a groupId (e.g., my.company)
    # Define an artifactId (e.g., my.company.do_something)
    # Define a version (e.g., 1.0.0-SNAPSHOT)
    # Define a package (e.g., my.company.do_something.plugin)
  cd my.company.do_something
  mvn verify

After running this sequence of commands, we should see the eclipse workbench open and successfully run the canary integration test for our test project.

The generated project structure

Dynamic Directories

On closer comparison of the directory tree from the original project and the generated project, you might wonder how Maven knows to name folders based on the artifactIds.
If we look at the generated archetype, we will see a parameterized directory structure.

Parameterized directories are in red

You might also notice that the Activator.java and ActivatorTest.java are contained in the root of their src/main/java and src/test/java folders, respectively, in the archetype resources; however, in the generated project, the files are in the proper subdirectories with corresponding package declarations. In the generated archetype-metadata.xml, the contents of these source directories are filtered and packaged.

src/main/resources/META-INF/maven/archetype-metadata.xml

<archetype-descriptor ....>
  ....
  <modules>
    <module id="${rootArtifactId}.plugins" ....>
      <modules>
        <module id="${rootArtifactId}.plugin" ....>
          <fileSets>
            <fileSet filtered="true" packaged="true" encoding="UTF-8">
              <directory>src/main/java</directory>
              <includes>
                <include>**/*.java</include>
              </includes>
            </fileSet>
            ....
          </fileSets>
        </module>
        <module id="${rootArtifactId}.plugin.tests" ....>
          <fileSets>
            <fileSet filtered="true" packaged="true" encoding="UTF-8">
              <directory>src/test/java</directory>
              <includes>
                <include>**/*.java</include>
              </includes>
            </fileSet>
            ....
          </fileSets>
        </module>
      </modules>
    </module>
    ....
  </modules>
</archetype-descriptor>

Testing

After customizing our resources and adding any additional required properties, we can integration-test the archetype using a golden template project contained in the src/test/projects/basic/reference directory. For this example, if the generated my.company.do_something project above exactly meets our needs, then we can simply copy the entire contents of this directory to the src/test/projects/basic/reference folder and update the archetype.properties to match the properties we used for project generation.

src/test/resources/projects/basic/archetype.properties

groupId=my.company
artifactId=my.company.do_something
version=1.0.0-SNAPSHOT
package=my.company.do_something.plugin

During the integration-test phase, Maven will generate a sample project from the archetype with these properties and will compare the results with the golden template.

mvn verify

NB: use of this type of golden template can be very brittle, and any changes made to files in the src/main/resources/archetype-resources directory must be reflected by equivalent changes in the src/test/resources/projects/basic/reference directory.

Publish to a Maven Repository

Now that our Maven archetype is tested and ready for publication, we can publish it, either to an internal repository, to the sonatype repository, which will synch our artifact with Maven central, or to a private repository, which in our case can simply be github pages.
We want to partition our repository into snapshots and releases, and we will need archetype catalogs in a subfolder of each.
We can deploy our archetype to a folder that contains the gh-pages branch of a github project called maven, which will contain all our published maven artifacts.

mvn -DaltDeploymentRepository=snapshot-repo::default::file:/path/to/git_maven_repository/snapshots clean deploy

Since we have installed the archetype, we already have a ~/.m2/archetype-catalog.xml file.

<?xml version="1.0" encoding="UTF-8"?>
<archetype-catalog
  xsi:schemaLocation="http://maven.apache.org/plugins/maven-archetype-plugin/archetype-catalog/1.0.0 http://maven.apache.org/xsd/archetype-catalog-1.0.0.xsd"
  xmlns="http://maven.apache.org/plugins/maven-archetype-plugin/archetype-catalog/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <archetypes>
    <archetype>
      <groupId>timezra.eclipse</groupId>
      <artifactId>timezra.eclipse.save_actions_extensions-archetype</artifactId>
      <version>1.0.0-SNAPSHOT</version>
      <description>Parent project for the timezra.eclipse.save_actions_extensions project set</description>
    </archetype>
  </archetypes>
</archetype-catalog>

The Maven repository

Use the Archetype From the Repository

After our artifacts have been deployed to a Maven repository, we can generate a project from the deployed archetype in a way similar to how we generated the project from the archetype installed locally above.

Generating from an artifact in a gh-pages repository

mvn archetype:generate -DarchetypeCatalog=http://your_github_username.github.com/maven/snapshots/archetypes

Conclusion

This tutorial is a continuation of a previous post, in which we created a set of Tycho-enabled Eclipse projects. The archetype we created from this set of projects abstracts the boilerplate and conventions into a re-usable component that will make setting up similar projects even faster in the future.
The end result of this effort is the tycho_new_plugin_project.

Wednesday, October 12, 2011

Eclipse Plug-ins: Program to Publish

Goal

The purpose of this blog entry is to demonstrate one workflow for taking an idea for an Eclipse plug-in from concept to product downloadable from the Eclipse marketplace.

tl;dr

If you want to look right away at working code for a simple Eclipse plug-in, feature, update site and their Tycho configurations, a sample project based on this tutorial is available. The plug-in can also be installed in Eclipse through the Marketplace (search for "save actions") or directly from an update site. In addition, the conventions for project setup contained in this tutorial have been extracted into a Maven archetype. Since this blog entry does not provide an in-depth introduction to the nuances of configuring and writing Eclipse plug-ins, having the project source as a reference, either locally or in a web browser, will be helpful for following the examples.

The Idea

Suppose we are starting a new project on an existing codebase and, as part of our Working Agreement, the team has decided to automate certain coding standards with Eclipse Java editor save actions. It would be convenient to bring all the existing code up to our standards before development even begins. Currently, formatting and import organization can be performed in bulk from the source submenu in the workbench. Eclipse users can also apply clean-up conventions from this submenu. Unfortunately, even though the clean-up and save participants have similar configurations, there is a disconnect between the two.
Exposing save actions for the bulk processing of Java/Groovy files through this source submenu is a simple enough feature to add to Eclipse and should provide the opportunity to experience one full cycle of Eclipse plug-in development, from the creation of a simple plug-in and its integration tests, to the addition of a feature to contain the plug-in, to the packaging of an update site, to the distribution of the product through the Eclipse Marketplace.

The Plug-in

We will begin by creating a new plug-in project named timezra.eclipse.apply_save_actions in our Eclipse workspace. Since we will eventually generate a Tycho configuration in order to automate the compilation, testing and packaging of our product, we will modify a few of the default settings for the plug-in. Our plug-in and fragment projects will be contained in a plugins subdirectory, here /path/to/workspace/plugins/timezra.eclipse.apply_save_actions. Our source folder will be src/main/java and output folder will be target/classes to follow the Maven convention.

The New Plug-in Project wizard with Maven-inspired configurations

We can either use the New Plug-in Project Wizard Hello, World Command Template or we can contribute a command to a menu with manual configuration. There are already quite a few resources for contributing commands through an extension point, so you can get more insight into specific configuration settings there.
For this example, the configuration is boilerplate. The internals of the ApplySaveActions command handler are somewhat interesting. There may be a more direct way to invoke the save participant, but this method works on both un-opened files and source buffers modified in memory for Java and Groovy, so in the interest of DTSTTCPW, we can use this simple implementation until it is no longer sufficient.

plugins/timezra.eclipse.apply_save_actions/src/main/java/timezra/eclipse/apply_save_actions/handlers/ApplySaveActions.java

package timezra.eclipse.apply_save_actions.handlers;

import static java.util.Arrays.asList;
import java.lang.reflect.InvocationTargetException;
import java.util.ArrayList;
import java.util.Collection;
import org.eclipse.core.commands.AbstractHandler;
import org.eclipse.core.commands.ExecutionEvent;
import org.eclipse.core.commands.ExecutionException;
import org.eclipse.core.resources.IFile;
import org.eclipse.core.resources.IWorkspace;
import org.eclipse.core.resources.ResourcesPlugin;
import org.eclipse.core.runtime.CoreException;
import org.eclipse.core.runtime.IAdapterManager;
import org.eclipse.core.runtime.IProgressMonitor;
import org.eclipse.core.runtime.OperationCanceledException;
import org.eclipse.core.runtime.Platform;
import org.eclipse.jdt.core.ICompilationUnit;
import org.eclipse.jdt.core.IJavaElement;
import org.eclipse.jdt.core.IJavaProject;
import org.eclipse.jdt.core.IPackageFragment;
import org.eclipse.jdt.core.IPackageFragmentRoot;
import org.eclipse.jdt.core.JavaModelException;
import org.eclipse.jface.operation.IRunnableWithProgress;
import org.eclipse.jface.viewers.ISelection;
import org.eclipse.jface.viewers.IStructuredSelection;
import org.eclipse.ui.IWorkbench;
import org.eclipse.ui.PlatformUI;
import org.eclipse.ui.actions.WorkspaceModifyOperation;
import org.eclipse.ui.handlers.HandlerUtil;
import org.eclipse.ui.part.FileEditorInput;
import org.eclipse.ui.texteditor.IDocumentProvider;

public class ApplySaveActions extends AbstractHandler {

  private final IAdapterManager adapterManager;
  private final IWorkspace workspace;
  private final IWorkbench workbench;

  public ApplySaveActions() {
    this(Platform.getAdapterManager(), ResourcesPlugin.getWorkspace(), PlatformUI.getWorkbench());
  }

  ApplySaveActions(final IAdapterManager adapterManager, final IWorkspace workspace, final IWorkbench workbench) {
    this.adapterManager = adapterManager;
    this.workspace = workspace;
    this.workbench = workbench;
  }

  @Override
  public Object execute(final ExecutionEvent event) throws ExecutionException {
    final ISelection currentSelection = HandlerUtil.getCurrentSelectionChecked(event);
    if (currentSelection instanceof IStructuredSelection) {
      final IStructuredSelection selections = (IStructuredSelection) currentSelection;
      try {
        applyTo(selections);
      } catch (final JavaModelException e) {
        throw new ExecutionException(Messages.APPLY_SAVE_ACTIONS_UNEXPECTED_ERROR, e);
      } catch (final InvocationTargetException e) {
        throw new ExecutionException(Messages.APPLY_SAVE_ACTIONS_UNEXPECTED_ERROR, e.getTargetException());
      }
    }
    return null;
  }

  private void applyTo(final IStructuredSelection selections) throws JavaModelException, InvocationTargetException {
    for (final Object o : selections.toList()) {
      final IJavaProject javaProject = getAdapter(o, IJavaProject.class);
      if (javaProject != null) {
        applyTo(javaProject.getPackageFragments());
        continue;
      }
      final IPackageFragmentRoot packageFragmentRoot = getAdapter(o, IPackageFragmentRoot.class);
      if (packageFragmentRoot != null) {
        applyTo(packageFragmentRoot);
        continue;
      }
      final IPackageFragment packageFragment = getAdapter(o, IPackageFragment.class);
      if (packageFragment != null) {
        applyTo(packageFragment);
        continue;
      }
      final ICompilationUnit compilationUnit = getAdapter(o, ICompilationUnit.class);
      if (compilationUnit != null) {
        applyTo(compilationUnit);
        continue;
      }
    }
  }

  private void applyTo(final IPackageFragmentRoot packageFragmentRoot) throws JavaModelException,
      InvocationTargetException {
    final IJavaElement[] children = packageFragmentRoot.getChildren();
    final IPackageFragment[] fragments = new IPackageFragment[children.length];
    System.arraycopy(children, 0, fragments, 0, children.length);
    applyTo(fragments);
  }

  private void applyTo(final IPackageFragment... packageFragments) throws JavaModelException,
      InvocationTargetException {
    final Collection<ICompilationUnit> compilationUnits = new ArrayList<ICompilationUnit>();
    for (final IPackageFragment f : packageFragments) {
      compilationUnits.addAll(asList(f.getCompilationUnits()));
    }
    applyTo(compilationUnits.toArray(new ICompilationUnit[compilationUnits.size()]));
  }

  private void applyTo(final ICompilationUnit... compilationUnits) throws InvocationTargetException {
    final IRunnableWithProgress delegate = new ApplySaveActionsOperation(compilationUnits);
    try {
      workbench.getProgressService().run(false, true, delegate);
    } catch (final InterruptedException e) {
      // cancellation is fine
    }
  }

  @SuppressWarnings("restriction")
  private IDocumentProvider createDocumentProvider() {
    return new org.eclipse.jdt.internal.ui.javaeditor.CompilationUnitDocumentProvider();
  }

  @SuppressWarnings("unchecked")
  private <T> T getAdapter(final Object o, final Class<T> c) {
    return (T) adapterManager.getAdapter(o, c);
  }

  private final class ApplySaveActionsOperation extends WorkspaceModifyOperation {
    private final ICompilationUnit[] compilationUnits;

    ApplySaveActionsOperation(final ICompilationUnit... compilationUnits) {
      this.compilationUnits = compilationUnits;
    }

    @Override
    public void execute(final IProgressMonitor pm) throws CoreException {
      pm.beginTask(Messages.APPLY_SAVE_ACTIONS_BEGIN_TASK, compilationUnits.length);
      try {
        for (final ICompilationUnit unit : compilationUnits) {
          applyTo(workspace.getRoot().getFile(unit.getPath()), pm);
        }
      } finally {
        pm.done();
      }
    }

    void applyTo(final IFile f, final IProgressMonitor pm) throws CoreException {
      report(f.getName(), pm);
      final IDocumentProvider provider = createDocumentProvider();
      final FileEditorInput editorInput = new FileEditorInput(f);
      try {
        provider.connect(editorInput);
        provider.aboutToChange(editorInput);
        provider.saveDocument(pm, editorInput, provider.getDocument(editorInput), true);
      } finally {
        provider.changed(editorInput);
        provider.disconnect(editorInput);
      }
    }

    void report(final String task, final IProgressMonitor pm) {
      if (pm.isCanceled()) {
        throw new OperationCanceledException();
      }
      pm.setTaskName(task);
      pm.worked(1);
    }
  }
}

The Test Fragment

We will similarly create a new integration test fragment alongside this plug-in, overriding the default configuration to store the fragment into the plugins subdirectory and to use Maven conventions for the source and output directories.

The New Fragment Project wizard with Maven-inspired configurations

The New Fragment Project wizard with Maven-inspired configurations

There are various approaches for testing Eclipse plug-ins, but the fragment approach has been embraced by the Tycho community, so we will use it here.
Again, the project configuration can be boilerplate for now. Of particular interest is the handler test case.

plugins/timezra.eclipse.apply_save_actions.tests/src/test/java/timezra/eclipse/apply_save_actions/handlers/ApplySaveActions.java

package timezra.eclipse.apply_save_actions.handlers;

import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNull;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Scanner;
import org.eclipse.core.commands.ExecutionEvent;
import org.eclipse.core.commands.ExecutionException;
import org.eclipse.core.expressions.EvaluationContext;
import org.eclipse.core.resources.IContainer;
import org.eclipse.core.resources.IFile;
import org.eclipse.core.resources.IFolder;
import org.eclipse.core.resources.IProject;
import org.eclipse.core.resources.IResource;
import org.eclipse.core.resources.IWorkspaceRoot;
import org.eclipse.core.resources.ResourcesPlugin;
import org.eclipse.core.runtime.CoreException;
import org.eclipse.core.runtime.IProgressMonitor;
import org.eclipse.core.runtime.NullProgressMonitor;
import org.eclipse.core.runtime.Path;
import org.eclipse.core.runtime.SubProgressMonitor;
import org.eclipse.core.runtime.preferences.InstanceScope;
import org.eclipse.jdt.core.IClasspathEntry;
import org.eclipse.jdt.core.IJavaProject;
import org.eclipse.jdt.core.JavaCore;
import org.eclipse.jdt.launching.JavaRuntime;
import org.eclipse.jdt.ui.JavaUI;
import org.eclipse.jdt.ui.cleanup.CleanUpOptions;
import org.eclipse.jface.text.TextSelection;
import org.eclipse.jface.viewers.StructuredSelection;
import org.eclipse.ui.ISources;
import org.junit.After;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.MethodRule;
import timezra.eclipse.apply_save_actions.Constants;
import timezra.eclipse.apply_save_actions.tests.ModifiesSaveActionsPreferences;
import timezra.eclipse.apply_save_actions.tests.ModifiesSaveActionsPreferencesRule;

public class ApplySaveActionsPluginTest {

  private static final String SOURCE_FOLDER = "src/test/java";

  private static final String EOL = System.getProperty("line.separator");

  private static final IProgressMonitor NULL_PROGRESS_MONITOR = new NullProgressMonitor();

  private static final String TEST_CLASS = "TestClass";
  private static final String TEST_PACKAGE = "timezra.eclipse.apply_save_actions";

  private static final String TEST_CLASS_BEFORE_SAVE_ACTIONS = "package " + TEST_PACKAGE
      + ";import java.util.*;class " + TEST_CLASS + "{private List<" + TEST_CLASS + "> l;" + TEST_CLASS
      + "(List<" + TEST_CLASS + "> l){this.l=l;}}";

  private static final String TEST_CLASS_AFTER_SAVE_ACTIONS = "package " + TEST_PACKAGE + ";" + EOL + //
      EOL + //
      "import java.util.List;" + EOL + //
      EOL + //
      "class " + TEST_CLASS + " {" + EOL + //
      "  private final List<" + TEST_CLASS + "> l;" + EOL + //
      EOL + //
      "  " + TEST_CLASS + "(List<" + TEST_CLASS + "> l) {" + EOL + //
      "    this.l = l;" + EOL + //
      "  }" + EOL + //
      "}";

  @Rule
  public final MethodRule rule = new ModifiesSaveActionsPreferencesRule();

  private IProject aJavaProject;
  private IFolder aJavaPackage;
  private IFile aJavaFile;

  private IFolder aJavaSourceFolder;

  @Before
  public void setUp() throws CoreException {
    aJavaProject = createAJavaProject("a_java_project");
    aJavaSourceFolder = createASourceFolder(SOURCE_FOLDER);
    aJavaPackage = createAPackage(aJavaSourceFolder, TEST_PACKAGE.replaceAll("\\.", "/"));
    aJavaFile = createAJavaFile(aJavaPackage, TEST_CLASS + ".java");
  }

  @After
  public void tearDown() throws CoreException {
    aJavaProject.delete(true, NULL_PROGRESS_MONITOR);
  }

  @Test
  public void theCurrentSelectionMustBeStructured() throws ExecutionException {
    final ApplySaveActions command = new ApplySaveActions();
    final EvaluationContext context = new EvaluationContext(null, new Object());
    context.addVariable(ISources.ACTIVE_CURRENT_SELECTION_NAME, new TextSelection(0, 100));
    final ExecutionEvent event = new ExecutionEvent(null, Collections.emptyMap(), null, context);
    assertNull(command.execute(event));
  }

  @Test
  @ModifiesSaveActionsPreferences
  public void aJavaFileCanBeReformatted() throws ExecutionException, CoreException, IOException {
    enableJavaSaveActions();

    applySaveActions(JavaCore.create(aJavaFile));

    verifyThatSaveActionsHaveBeenApplied(aJavaFile);
  }

  @Test
  @ModifiesSaveActionsPreferences
  public void aJavaPackageCanBeReformatted() throws ExecutionException, CoreException, IOException {
    enableJavaSaveActions();

    applySaveActions(JavaCore.create(aJavaPackage));

    verifyThatSaveActionsHaveBeenApplied(aJavaFile);
  }

  @Test
  @ModifiesSaveActionsPreferences
  public void aJavaSourceFolderCanBeReformatted() throws ExecutionException, CoreException, IOException {
    enableJavaSaveActions();

    applySaveActions(JavaCore.create(aJavaSourceFolder));

    verifyThatSaveActionsHaveBeenApplied(aJavaFile);
  }

  @Test
  @ModifiesSaveActionsPreferences
  public void aJavaProjectCanBeReformatted() throws ExecutionException, CoreException {
    enableJavaSaveActions();

    applySaveActions(JavaCore.create(aJavaProject));

    verifyThatSaveActionsHaveBeenApplied(aJavaFile);
  }

  private void applySaveActions(final Object selection) throws ExecutionException {
    final ApplySaveActions command = new ApplySaveActions();
    final EvaluationContext context = new EvaluationContext(null, new Object());
    context.addVariable(ISources.ACTIVE_CURRENT_SELECTION_NAME, new StructuredSelection(selection));
    final ExecutionEvent event = new ExecutionEvent(null, Collections.emptyMap(), null, context);
    command.execute(event);
  }

  // contains a beaut that turns a stream into a String without using IoUtils:
  // http://stackoverflow.com/questions/309424/in-java-how-do-a-read-convert-an-inputstream-in-to-a-string
  private void verifyThatSaveActionsHaveBeenApplied(final IFile aJavaFile) throws CoreException {
    final String actualContents;
    final Scanner scanner = new Scanner(aJavaFile.getContents());
    try {
      actualContents = scanner.useDelimiter("\\A").next();
    } finally {
      scanner.close();
    }
    assertEquals(TEST_CLASS_AFTER_SAVE_ACTIONS, actualContents);
  }

  @SuppressWarnings("restriction")
  private void enableJavaSaveActions() {
    InstanceScope.INSTANCE.getNode(JavaUI.ID_PLUGIN).putBoolean(Constants.PERFORM_SAVE_ACTIONS_PREFERENCE, true);

    final Map<String, String> cleanupPreferences = new HashMap<String, String>(
        org.eclipse.jdt.internal.ui.JavaPlugin
            .getDefault()
            .getCleanUpRegistry()
            .getDefaultOptions(
                org.eclipse.jdt.internal.corext.fix.CleanUpConstants.DEFAULT_SAVE_ACTION_OPTIONS)
            .getMap());

    cleanupPreferences.put(org.eclipse.jdt.internal.corext.fix.CleanUpConstants.FORMAT_SOURCE_CODE,
        CleanUpOptions.TRUE);
    cleanupPreferences.put(org.eclipse.jdt.internal.corext.fix.CleanUpConstants.ORGANIZE_IMPORTS,
        CleanUpOptions.TRUE);
    cleanupPreferences.put(org.eclipse.jdt.internal.corext.fix.CleanUpConstants.CLEANUP_ON_SAVE_ADDITIONAL_OPTIONS,
        CleanUpOptions.TRUE);
    org.eclipse.jdt.internal.corext.fix.CleanUpPreferenceUtil.saveSaveParticipantOptions(InstanceScope.INSTANCE,
        cleanupPreferences);
  }

  private IFolder createASourceFolder(final String name) throws CoreException {
    final IFolder aJavaSourceFolder = aJavaProject.getFolder(Path.fromPortableString(name));
    create(aJavaSourceFolder);
    return aJavaSourceFolder;
  }

  private IFolder createAPackage(final IFolder aJavaSourceFolder, final String name) throws CoreException {
    final IFolder aJavaPackage = aJavaSourceFolder.getFolder(Path.fromPortableString(name));
    create(aJavaPackage);
    return aJavaPackage;
  }

  private void create(final IFolder folder) throws CoreException {
    final IContainer parent = folder.getParent();
    if (parent.getType() == IResource.FOLDER && !parent.exists()) {
      create((IFolder) parent);
    }
    folder.create(true, true, NULL_PROGRESS_MONITOR);
  }

  private IFile createAJavaFile(final IFolder aJavaPackage, final String name) throws CoreException {
    final IFile aJavaFile = aJavaPackage.getFile(Path.fromPortableString(name));
    aJavaFile.create(new ByteArrayInputStream(TEST_CLASS_BEFORE_SAVE_ACTIONS.getBytes()), true,
        NULL_PROGRESS_MONITOR);
    return aJavaFile;
  }

  // based on http://www.stateofflow.com/journal/66/creating-java-projects-programmatically
  @SuppressWarnings("restriction")
  private IProject createAJavaProject(final String name) throws CoreException {
    final IWorkspaceRoot root = ResourcesPlugin.getWorkspace().getRoot();
    final IProject project = root.getProject(name);
    project.create(NULL_PROGRESS_MONITOR);
    project.open(NULL_PROGRESS_MONITOR);
    org.eclipse.jdt.internal.ui.wizards.buildpaths.BuildPathsBlock.addJavaNature(project, new SubProgressMonitor(
        NULL_PROGRESS_MONITOR, 1));
    final IJavaProject javaProject = JavaCore.create(project);

    final List<IClasspathEntry> entries = new ArrayList<IClasspathEntry>();
    for (final IClasspathEntry entry : javaProject.getRawClasspath()) {
      if (entry.getEntryKind() == IClasspathEntry.CPE_SOURCE) {
        ((org.eclipse.jdt.internal.core.ClasspathEntry) entry).path = Path.fromPortableString(SOURCE_FOLDER);
      }
      entries.add(entry);
    }
    entries.add(JavaRuntime.getDefaultJREContainerEntry());
    javaProject.setRawClasspath(entries.toArray(new IClasspathEntry[entries.size()]), NULL_PROGRESS_MONITOR);

    return project;
  }
}

The Feature

Now that we have a tested plug-in, we will create an Eclipse feature to contain it for distribution. We can do practically all our configuration through the New Feature Project Wizard, except we want to put the feature project into a features subdirectory in the same way that we put our plug-in and fragment into the plugins subdirectory.

The Update Site

Now that we have a tested plug-in and a feature to contain it, we will create an Eclipse update site for publishing the feature. In the New Update Site Project Wizard, we will again override the default location so that our update site project is in an update-site subdirectory, just as we separated our plug-in, fragment and feature into plugins and features subdirectories.

NB: Whereas we may have multiple features or plug-ins for our project, we will have a single update site; thus the update-site project is not contained in a subfolder of update-site, but in this folder directly.

Tycho

Compiling, running integration tests and packaging an application entirely within an IDE does not scale to even a single programmer over time, let alone a team of programmers working on multiple plug-in projects. So far, the amount of ceremony for creating our menu contribution has been high, but the IDE has reduced a significant amount of the boilerplate. Before Tycho, the amount of ceremony and hackery required to get plugins, features and update sites packaged and unit and integration test suites running on a CI server had been prohibitively high. Tycho removes a significant amount of that pain.
Generating meaningful poms for our projects is as trivial as going into each of the features, plugins and update-site directories and running a Tycho goal.

mvn org.eclipse.tycho:tycho-pomgenerator-plugin:generate-poms -DgroupId=timezra.eclipse

We can combine some of the boilerplate in each of the subproject poms in an über-parent pom at the root of our workspace. We will also add the Indigo p2 repository and a target platform configuration resolver since we are developing our Eclipse components Manifest-first.

pom.xml

We will also configure Tycho to use the UI test runner for our integration test suite, as well as add any platform-specific runtime plug-in dependencies or configuration to the test fragment pom.

plugins/timezra.eclipse.apply_save_actions.tests/pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"
  xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <modelVersion>4.0.0</modelVersion>
  <parent>
    <groupId>timezra.eclipse</groupId>
    <artifactId>apply-save-actions-plugins-parent</artifactId>
    <version>1.0.0-SNAPSHOT</version>
  </parent>
  <groupId>timezra.eclipse</groupId>
  <artifactId>timezra.eclipse.apply_save_actions.tests</artifactId>
  <version>1.0.0-SNAPSHOT</version>
  <packaging>eclipse-test-plugin</packaging>
  
  <build>
    <outputDirectory>target/test-classes</outputDirectory>
    <plugins>
      <plugin>
        <groupId>org.eclipse.tycho</groupId>
        <artifactId>tycho-surefire-plugin</artifactId>
        <version>${tycho-version}</version>
        <configuration>
          <useUIHarness>true</useUIHarness>
        </configuration>
      </plugin>
    </plugins>
  </build>
  <profiles>
    <profile>
      <id>osx</id>
      <activation>
        <property>
          <name>java.vendor.url</name>
          <value>http://www.apple.com/</value>
        </property>
      </activation>
      <build>
        <pluginManagement>
          <plugins>
            <plugin>
              <groupId>org.eclipse.tycho</groupId>
              <artifactId>tycho-surefire-plugin</artifactId>
              <version>${tycho-version}</version>
              <configuration>
                <argLine>-XstartOnFirstThread</argLine>
                <dependencies>
                  <dependency>
                    <type>p2-installable-unit</type>
                    <artifactId>org.eclipse.jdt.launching.macosx</artifactId>
                  </dependency>
                </dependencies>
              </configuration>
            </plugin>
          </plugins>
        </pluginManagement>
      </build>
    </profile>
  </profiles>
</project>

NB: we would use a different configuration if our fragment contained unit-tests instead of integration tests.

JAR signing

If you do not have access to a certificate from a trusted authority, you can generate a self-signed certificate with 1-year validity by a command such as

keytool -genkey -alias _keystore_alias_ -keystore /path/to/keystore -validity 365

In order to sign the jars deployed to our update-site before release, we will add a new profile with a plug-in management configuration to the über-parent pom.

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project ....>
  ....
  <profiles>
    <profile>
      <id>sign</id>
      
      <build>
        <pluginManagement>
          <plugins>
            <plugin>
              <groupId>org.apache.maven.plugins</groupId>
              <artifactId>maven-jarsigner-plugin</artifactId>
              <version>1.2</version>
              <executions>
                <execution>
                  <goals>
                    <id>sign</id>
                    <goal>sign</goal>
                  </goals>
                </execution>
                <execution>
                  <goals>
                    <id>verify</id>
                    <goal>verify</goal>
                  </goals>
                </execution>
              </executions>
            </plugin>
          </plugins>
        </pluginManagement>
      </build>
    </profile>
  </profiles>
  ....
</project>

Similarly, we will configure the plug-ins and features parent poms for jar signing.

plugins/pom.xml and features/pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project ....>
  ....
  <profiles>
    <profile>
      <id>sign</id>
      <build>
        <plugins>
          <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-jarsigner-plugin</artifactId>
          </plugin>
        </plugins>
      </build>
    </profile>
  </profiles>
</project>

Publishing

We are now ready to build and test our signed plug-ins and features and to package them in an update site for deployment.

mvn -Psign -Djarsigner.keystore=/path/to/keystore -Djarsigner.storepass=_keystore_password_ -Djarsigner.alias=_keystore_alias_ clean package integration-test

There will now be a fully deployable update site in update-site/target/site.
For this particular project, I distribute the contents of this directory to the gh-pages branch of the project on github.
Since the update site for the project is publicly-available and since I have an Eclipse Bugzilla login, I can simply add a new solution listing to the Eclipse Marketplace to make the menu contribution even more discoverable by and accessible to Eclipse users.

Conclusion

This tutorial has provided the outline for the workflow of taking an idea for a single-featured Eclipse contribution from inception to delivery in a very short amount of time. While Eclipse tooling has for years provided the means to perform the tasks of plug-in, fragment, feature and update site publishing entirely within the IDE, it is the Tycho project that lowers the barrier to entry for scaling out plug-in development by making the build and test process far simpler to automate and to configure than other PDE-based build systems. Along the way, we have explored JAR signing and uploading of content to github and to the Eclipse Marketplace, and we have hopefully developed a menu contribution that others will find useful in their own projects.

The source submenu contribution for applying save actions in batch mode

The source submenu contribution for applying save actions in batch mode

Overcoming Fear

About Me

Blog Archive

Other XP Bloggers

Thursday, November 10, 2011

Trampoline and Memoize

Goal

tl;dr

Simple Memoization

Recursive Memoization

Trampoline

Memoizing a Trampolined Closure

Trampolining a Memoized Closure

Conclusion

Wednesday, October 19, 2011

Tycho Archetype

Goal

tl;dr

Generate Archetype From Project

Dynamic Directories

Testing

Publish to a Maven Repository

Use the Archetype From the Repository

Conclusion

Wednesday, October 12, 2011

Eclipse Plug-ins: Program to Publish

Goal

tl;dr

The Idea

The Plug-in

The Test Fragment

The Feature

The Update Site

Tycho

JAR signing

Publishing

Conclusion