Imperceptible Thoughts

Yet Another Blog Hosting Change

Wed, 05 Feb 2014 00:00:00 -0600

Only a few months ago, I decided to switch my personal blog (which isn’t maintained very well)
from a privately hosted Wordpress installation on a server in my basement to Tumblr. There
were a lot of reasons for this but the biggest was, I’m not very interested in maintaining that
server anymore.

Tumblr has ben ok. But I’ve found that, I don’t like any blogging platform where I have to link
to an external page to get source code included in the blog post (lame/too much effort). The
obvious choice to correct this was to use something based on Markdown so I can write code
inline. And that inevitably led me to using Github Pages.

That’s all well and good, but Github Pages doesn’t provided any type of default scheming, you
are responsible for everything. That’s cool if you’re into that kind of stuff, but CSS and
skinning has never been a strong suite of mine. So, I needed something with some generic, yet
good looking templates. In comes Octopress.

I’m just starting to learn the ropes here, so I’ll be doing a lot to play with this platform
before finally switching the DNS record over, but so far I’m happy.

More to come!

Grails-Gradle Plugin 2.0.0 Released

Mon, 27 Jan 2014 00:00:00 -0600

I am happy to announce that the 2.0.0 release for the Grails-Gradle plugin has finally been released.

This release has been a long time coming, and offers some significant improvements over the 1.1.0 release. Additionally, many people have been relying on the 2.0.0-SNAPSHOT release and this locks-in a stable version that is now available in Maven Central and the Grails Central repositories.

ChangeLog

Automatic configuration of Grails repositories using grails.central() in your repositories block
Automatic configuration of Grails, Groovy, and Spring Loaded versions
Implementation of a build task structure more similar to a standard Java project (i.e. clean, build, assemble, test, check ,etc.)
Better integration with IntelliJ IDEA (plugin source is now available)
Definition of Grails source sets to allow for task input/output configuration
Various bug fixes and improvements in Grails command spawning.

Additional information can be found on the plugin’s Github page.

Reproducible Multi-Project Gradle Builds (Part 2)

Fri, 20 Dec 2013 00:00:00 -0600

Follow up to my previous post - Reproducible Multi-Project Gradle builds

I got around to doing some testing on this using a simple Gradle project. Very quickly did I find out that this particular strategy was not going to be easy…and potentially impossible.

Problem 1 - Setting Timestamp value on Jar Entry’s

The first problem I ran into was actually setting the timestamp value of a Jar Entry. My first attempt looked something like this:

jar {
  doLast {
    long ts = fileTree('.') {
      exclude 'build/**'
    }.files.sort {
      -(it.lastModified())
    }.first().lastModified()
    JarFile jf = new JarFile(archivePath)
    jf.entries().each { entry ->
      entry.time = ts
    }
    jf.close()
  }
}

Yup. That didn’t work. Jar’s are more or less read-only. So in order to change the time stamp, I needed to write out a new Jar file with the modified entries. Basically, create a new jar file, iterate over the entries from the first jar file, clone them, modified the time stamp, add the cloned entry to the new jar file and then binary copy the data from the first jar file to the second. It looks something like this:

jar {
  doLast {
    long ts = fileTree('.') {
      exclude 'build/**'
    }.files.sort {
      -(it.lastModified())
    }.first().lastModified()
    File newJar = new File(archivePath.parent, 'new-' + archiveName)
    JarOutputStream jos = new JarOutputStream(new FileOutputStream(newJar))
    JarFile jf = new JarFile(archivePath)
    jf.entries().each { entry ->
      cloneAndCopyEntry(jf, entry, jos, ts)
    }
    jos.finish()
    jf.close()
  }
}
 
void cloneAndCopyEntry(JarFile originalFile, JarEntry original, JarOutputStream jos, long newTimestamp) {
  JarEntry clone = new JarEntry(original)
  clone.time = newTimestamp
  def entryIs = originalFile.getInputStream(original)
  jos.putNextEntry(clone)
  copyBinaryData(entryIs, jos)
}
 
void copyBinaryData(InputStream is, JarOutputStream jos) {
  byte[] buffer = new byte[1024]
  int len = 0
  while((len = is.read(buffer)) != -1) {
    jos.write(buffer, 0, len)
  }
}

Problem 2 - JAR “Magic” Extra Byte

The next issue I encountered was that my new Jar file always had an entry that didn’t match the original Jar for it’s ‘extra’ property. JarEntry.geExtra() returns a byte[]. In the orignal file, this was null, but in my copied file it was set to some data. I finally found that it is from the implementation of JarOutputStream in Java.

This Jar MAGIC byte gets added to the first entry in the Jar file. I haven’t been able to find any documentation on what it is for, but a friend thought it was likely for the Jar tool itself to determine if an archive file is actually a Jar.

Curiously, Jar files produced by the Gradle Jar task, do NOT have this magic byte. Digging into their code, I found that they use a ZipOutputStream to write out the Jar file which doesn’t have this magic byte code. Using ZipOutputStream and ZipEntry works just as fine and avoids this, so we update the build to do the same:

jar {
  doLast {
    long ts = fileTree('.') {
      exclude 'build/**'
    }.files.sort {
      -(it.lastModified())
    }.first().lastModified()
    File newJar = new File(archivePath.parent, 'new-' + archiveName)
    ZipOutputStream zos = new ZipOutputStream(new FileOutputStream(newJar))
    JarFile jf = new JarFile(archivePath)
    jf.entries().each { entry ->
      cloneAndCopyEntry(jf, entry, zos, ts)
    }
    zos.finish()
    jf.close()
  }
}
 
void cloneAndCopyEntry(JarFile originalFile, JarEntry original, ZipOutputStream zos, long newTimestamp) {
  ZipEntry clone = new ZipEntry(original)
  clone.time = newTimestamp
  def entryIs = originalFile.getInputStream(original)
  zos.putNextEntry(clone)
  copyBinaryData(entryIs, zos)
}
 
void copyBinaryData(InputStream is, ZipOutputStream zos) {
  byte[] buffer = new byte[1024]
  int len = 0
  while((len = is.read(buffer)) != -1) {
    zos.write(buffer, 0, len)
  }
}

Problem 3 - Zip timestamp spec

After all this, I still wasn’t getting subsequent builds of the Jar to have matching checksums. I wrote a method that at the end of producing the timestamp adjusted Jar iterated over all the entries and compared all the fields to the original except the time field which I compared to the expected time stamp. Looks like this:

jar {
  doLast {
    long ts = fileTree('.') {
      exclude 'build/**'
    }.files.sort {
      -(it.lastModified())
    }.first().lastModified()
    File newJar = new File(archivePath.parent, 'new-' + archiveName)
    ZipOutputStream zos = new ZipOutputStream(new FileOutputStream(newJar))
    JarFile jf = new JarFile(archivePath)
    jf.entries().each { entry ->
      cloneAndCopyEntry(jf, entry, zos, ts)
    }
    zos.finish()
    jf.close()
    compareJars(archivePath, newJar, ts)
  }
}
 
void compareJars(File original, File copy, long ts) {
  def jf = new JarFile(original)
  def cjf = new JarFile(copy)
  jf.entries().each { entry ->
    def centry = cjf.getJarEntry(entry.name)
    compareEntries(entry, centry, ts)
  }
}
 
void compareEntries(JarEntry entry, JarEntry centry, long ts) {
  assert entry.name == centry.name
  assert entry.comment == centry.comment
  assert entry.compressedSize == centry.compressedSize
  assert entry.crc == centry.crc
  assert entry.extra == centry.extra
  assert entry.method == centry.method
  assert entry.size == centry.size
  assert ts == centry.time
  assert entry.hashCode() == centry.hashCode()
}
 
void cloneAndCopyEntry(JarFile originalFile, JarEntry original, ZipOutputStream zos, long newTimestamp) {
  ZipEntry clone = new ZipEntry(original)
  clone.time = newTimestamp
  def entryIs = originalFile.getInputStream(original)
  zos.putNextEntry(clone)
  copyBinaryData(entryIs, zos)
}
 
void copyBinaryData(InputStream is, ZipOutputStream zos) {
  byte[] buffer = new byte[1024]
  int len = 0
  while((len = is.read(buffer)) != -1) {
    zos.write(buffer, 0, len)
  }
}

This showed me that some entries still didn’t have the same timestamp. Curiously, they were always off by 1 second (1000 milliseconds in the script). Digging back into the Java source code for ZipEntry, I found it was doing some sort of unix - DOS conversion and was losing the resolution of the time.

After a call out to the Twitterverse, a friend pointed out that the Zip Spec specifies that Zip entries have a time resolution of 2 seconds:

The update parameter controls what happens if the ZIP file already exists. When set to yes, the ZIP file is updated with the files specified. (New files are added; old files are replaced with the new versions.) When set to no (the default) the ZIP file is overwritten if any of the files that would be added to the archive are newer than the entries inside the archive. Please note that ZIP files store file modification times with a granularity of two seconds. If a file is less than two seconds newer than the entry in the archive, Apache Ant will not consider it newer.

So, we need to mimic this same behavior when producing our new Jar. Basically, we need to convert our timestamp to a resolution of 2 seconds. It looks like this:

jar {
  doLast {
    long ts = fileTree('.') {
      exclude 'build/**'
    }.files.sort {
      -(it.lastModified())
    }.first().lastModified()
    File newJar = new File(archivePath.parent, 'new-' + archiveName)
    ZipOutputStream zos = new ZipOutputStream(new FileOutputStream(newJar))
    JarFile jf = new JarFile(archivePath)
    jf.entries().each { entry ->
      long adjustedTs = convertTimestamp(ts)
      cloneAndCopyEntry(jf, entry, zos, adjustedTs)
    }
    zos.finish()
    jf.close()
    compareJars(archivePath, newJar, ts)
  }
}
 
long convertTimestamp(long timestamp) {
  long seconds = timestamp / 1000
  if (seconds % 2 != 0) {
    seconds += 1
  }
  return 1000 * seconds
}
 
void compareJars(File original, File copy, long ts) {
  def jf = new JarFile(original)
  def cjf = new JarFile(copy)
  long adjustedTs = convertTimestamp(ts)
  jf.entries().each { entry ->
    def centry = cjf.getJarEntry(entry.name)
    compareEntries(entry, centry, adjustedTs)
  }
}
 
void compareEntries(JarEntry entry, JarEntry centry, long ts) {
  assert entry.name == centry.name
  assert entry.comment == centry.comment
  assert entry.compressedSize == centry.compressedSize
  assert entry.crc == centry.crc
  assert entry.extra == centry.extra
  assert entry.method == centry.method
  assert entry.size == centry.size
  assert ts == centry.time
  assert entry.hashCode() == centry.hashCode()
}
 
void cloneAndCopyEntry(JarFile originalFile, JarEntry original, ZipOutputStream zos, long newTimestamp) {
  ZipEntry clone = new ZipEntry(original)
  clone.time = newTimestamp
  def entryIs = originalFile.getInputStream(original)
  zos.putNextEntry(clone)
  copyBinaryData(entryIs, zos)
}
 
void copyBinaryData(InputStream is, ZipOutputStream zos) {
  byte[] buffer = new byte[1024]
  int len = 0
  while((len = is.read(buffer)) != -1) {
    zos.write(buffer, 0, len)
  }
}

Problem 4 - Groovy `_timestamp` static field

This got my Jars closer, but they still weren’t checksumming the same. At this point, I started using a hex viewer to compare the two files. I used Hex Fiend because it can do a side by side diff of files and is free

Using a Hex viewer on a Jar file doesn’t do much since the data is compressed, but it can get you pointed in the right direction. In my cause I could see some byte differences around what appeared to be some class declarations

Next step was to explode the Jars and compare each of the files. This should a pretty apparent difference

There was a binary difference in the class file produced by subsequent compilations. The difference was related to a filed named __timestamp_239_neverHappen1387492963633

That’s an interesting field, because there’s nothing like that in my source file:

package john.app
 
import john.lib.EchoUtil
 
class EchoApp {
 
  public static void main(String[] args) {
    args.each {
      EchoUtil.echo(it)
    }
  }
}

Opening the .class file with a Java Decompiler (JD-GUI in this case), we see this for the class:

package john.app;
 
import groovy.lang.Closure;
import groovy.lang.GroovyObject;
import groovy.lang.MetaClass;
import john.lib.EchoUtil;
import org.codehaus.groovy.runtime.GeneratedClosure;
import org.codehaus.groovy.runtime.callsite.CallSite;
 
public class EchoApp
  implements GroovyObject
{
  public EchoApp()
  {
    EchoApp this;
    CallSite[] arrayOfCallSite = $getCallSiteArray();
    MetaClass localMetaClass = $getStaticMetaClass();
    this.metaClass = localMetaClass;
  }
 
  public static void main(String[] args)
  {
    CallSite[] arrayOfCallSite = $getCallSiteArray(); arrayOfCallSite[0].call(args, new _main_closure1(EchoApp.class)); }
  static { __$swapInit();
    long l1 = 0L;
    __timeStamp__239_neverHappen1387493347537 = l1;
    long l2 = 1387493347537L; }
  class _main_closure1 extends Closure implements GeneratedClosure { public _main_closure1(Object _thisObject) { super(_thisObject); }
    public Object doCall(Object it) { CallSite[] arrayOfCallSite = $getCallSiteArray(); return arrayOfCallSite[0].call(EchoUtil.class, it);
    }
 
    public Object doCall()
    {
      CallSite[] arrayOfCallSite = $getCallSiteArray();
      return doCall(null);
    }
 
    static
    {
      __$swapInit();
    }
  }
}

There is in fact a private variable being initialized. Opening the other copy of this class revealed that the name of this variable was changing:

__timeStamp__239_neverHappen1387493347537

vs.

__timeStamp__239_neverHappen1387492963633

Hmmm. That’s odd. A little more digging lead me to this Groovy-6308 - Timestamps in bytecode prevents baselining of code. The title of this bug is “Timestamps in bytecode prevents baselining of code”. Yup, this seems to be my problem.

It appears this field is injected as some sort of addition to SerialVersionUUID, though the corresponding conversations of that bug and the related bugs seems to indicate that it’s not really used for anything.

Unfortunately, the bugs are listed as being targeted for Groovy 3.0 which doesn’t help me in the near term.

At this point, since it’s a compiler function that’s stopping me, I’m not sure I have a path forward. This will probably get back burnered for a while since it’s not critical and more of a thought experiment than anything.

Reproducible Multi-Project Gradle Builds

Tue, 17 Dec 2013 00:00:00 -0600

Lately I’ve been working on a way to create a reproducible build of my application source using Gradle. And by reproducible, I mean, when I run the md5 or sha256 checksum on the resulting files, I get the same results.

This is a bit more complicate then I originally thought. The issue is that inherently, a Java Jar file includes the timestamp information for the files contained within. So, if you build a Jar and checksum it and then clean and rebuilt the Jar, even though the files are semantically identical, the resulting checksum’s will be different.

This is a problem in Continuous Integration/Deployment environments where you may be building from a clean VCS checkout every time but want to know only about the things that changed.

Working around this in Gradle is a bit of mind twister. After thinking about this for a couple days, I’ve come up with this list of items that need to addressed to successfully accomplish what I’m trying to do.

Calculate a single timestamp for each artifact that Gradle produces
Timestamp calculation needs to be sensitive to the Gradle project that produces the artifact.
Timestamp calculation needs to be sensitive to any Gradle project dependencies for the project producing the artifact.
Modify the resulting artifact to set the timestamp information for contents uniformly.

After a quick Twitter conversation with Luke Daley, item 4 was knocked out by using Java’s ZipFile API to iterate over the jar’s contents and set the lastModified timestamp to a value, but that leaves the more critical timestamp calculations.

Let’s start with a mind experiment to figure out what we need. Let’s assume we have the following Gradle project structure:

myApp
  |- utils
  |- app

This is a simple Gradle multi-project build where we have the application as one project and a utility library as another.

We check this code out and run a ./gradlew build and calculate an MD5 checksum on utils.jar and app.jar:

File	Checksum
utils.jar	A
app.jar	B

If I make a change to the app project and do a ./gradlew clean build, we want the following to happen:

File	Checksum
utils.jar	A
app.jar	B’

In this scenario, only the app.jar checksum would change because the utils project is functionally equivalent.

However, if we start from our initial commit and change the utils project and do a ./gradlew clean build, we want the following to happen:

File	Checksum
utils.jar	A’
app.jar	B’

In this case with only a change to utils, we want both checksums to change because the app project is dependent upon the changed code in the utils library.

My initial thought was to use my VCS to get a timestamp for the latest change to the Gradle project being built. This would couple my build to the VCS being used, but that’s an okay dependency in my opinion. Alternatively, the build could scan over the source contents and get the latest lastModified timestamp of any file contained within.

Neither of these solutions would solve the issue with upstream libraries however. Somehow, we need to propagate the upstream project’s information.

I’m not sure there is a nice way to do this, but my current thinking is to simulate this meta information by enforcing a commit to downstream projects when an upstream is modified. In my particular case, I would likely change the project’s version of the upstream library by modifying the build.gradle file in that project. I could add a task that verifies that the downstream project has a version that is at least equivalent to any upstream projects’ versions. This could run as part of the standard build process. This would force a change to the downstream’s build.gradle thus indirectly updating the timestamp the VCS would report for that project.

All of this is theoretical at this point, I’ve only started working on an implementation. I’ll update this post as I learn more.

Creating Self-contained, Executable Jars With Gradle and Shadow

Tue, 08 Oct 2013 13:43:54 -0500

Change of Heart...

Tue, 08 Oct 2013 00:00:00 -0500

As I was adding a new post today, I noticed an old post of mine that I had migrated from my old blog to this one about using IntelliJ and comparing it with STS. It’s a little over 2 years old and in it I make the statement “STS which is based on Eclipse, which I consider to be the premier Java IDE”;.

At the time, I was just starting my new job after 7 years with a large company where I primarily used Eclipse as my development tool on vanilla Java projects. And though I did really Eclipse at the time and was frustrated for quite a while with IntelliJ’s performance, I have to clarify that it’s is not my stance anymore.

Here’s the thing, I haven’t used Eclipse or STS for more than ~3mins in the last 2 1/2 years, so I cannot personally attest to its performance or usability. However, in the last 2 1/2 years, IntelliJ (I started with version 9 and am now using the 13 EAP) has come a long way. it’s performance is dramatically better. I’m faster using it now and I love the improvements with respect to Gradle integration they’ve been bringing in.

I’m not going to promote one over the other like I said because I don’t have enough usage of Eclipse recently to make a comparison and don’t really have the time or desire to look at it any time soon. So I’ll just leave it at that.

Profiling Grails Applications With VisualVM

Mon, 25 Mar 2013 08:55:00 -0500

Recovering From Merge Errors in Mercurial With Sub-Repositories and Named Branches

Mon, 25 Mar 2013 08:51:00 -0500

New Blog. Hooray

Mon, 25 Mar 2013 08:32:00 -0500

So I’ve decided to switch my blog over from a Wordpress sited hosted on a server in my basement to Tumblr. I don’t blog that much, but I want to do more and I don’t really want the responsibility of maintaining the server. Also, I want to keep my technical posts and my more mundane posts separate.

I’m going to republish some old posts of mine to this blog. I’m also going to republish some blog posts from my Object Partners blog here as well.

Contributing to Grails-Core With IntelliJ IDEA 11

Mon, 16 Jan 2012 00:00:00 -0600

I’ve recently started to work on contributing patches to the Grails Framework project. I’ve been lucky enough to be working for a client that is ok with using the latest versions of Grails for its back-end applications, so I’ve been able to play around a lot with Grails 2.0.0 since it’s release at the end of last year. However, this has resulted in me encountering a number of unexpected/irritating bugs that have sometimes prevented code from being migrated “as-is”. So, I’ve decided to try and learn and contribute back by working on some bugs myself.

If you’re looking to contribute yourself, start here http://grails.org/Contribute. I didn’t have any issues getting up and running with my own fork of grails-core. I did have to execute a ./gradlew compile task though before I was able to get any test to execute…and don’t worry, when this takes a while. There are A LOT of dependencies to download.

My next step was to start using IntelliJ 11 since it would allow me to quickly debug failing tests, so I again returned to the grails.org website and found the executing the ./gradlew idea task will produce the necessary IntelliJ project files…Excellent. But this is where the problems started.

First, I had to open the Module Settings for grails-script and remove the entry under ‘Source Folders’. This was causing IntelliJ to not save any edits I made to the module properties. It appears that this entry isn’t needed anyway (and I haven’t run into any issues with it yet).

Second, the project wouldn’t compile because the following 2 classes couldn’t be found:com.springsource.loaded.ReloadEventProcessorPlugin and com.springsource.loaded.Plugins. After some searching, I found the springloaded-core-1.0.2.jar in my Gradle User Home (~/.gradle) directory (look under ~/.gradle/caches/artifacts-7, it maybe be something different then artifact-7). I copied this jar to my home directory, opened the Module Settings for grails-core-grails-core and added a Jar Dependency that pointed to the springloaded-core-1.0.2.jar in my home directory.

Third, the grails-plugin-validation module won’t compile because it was missing a dependency to the grails-web module. Again, I opened the Module Settings for grails-plugin-validation and added a Module Dependency to grails-web.

Now, I can finally run and debug my tests and get working on some patches.