If I’m surprised about XML::Writer failing with old perls, CPAN Testers is happy to give me all the data it can to to look at trends. All the build results are available as a JSON file: for XML::Writer, that’s 5,749 builds going back to late 1999. (99.7% passes - well within the SLA for public domain software.)
For each test, I took the perl version used, the XML::Writer release being tested and when that test was run. I replaced the versions with release dates (actually, the commit date for that tag) and dragged it into R. So, over time, which perls are being used for tests? How far back does the Testers community go?
First, we have our dotted-line event horizon - no perl should be showing up in tests before release. The few escapees imply that I’m counting betas against the final release date. The plotting shows historical perls sticking around up to the present day, at least amongst the Testers participants. Overstriking symbols for points masks the weight of the upward trend, with more recent perls being far more common. Taking the mean test time for each release of perl, the blue line plots the line of best fit through those mean times. That is, the bulk of test runs are using a recent perl and the lag between release and widespread testing seems to be decreasing.
So, perls are generally current but with a long tail stretching back. How about XML::Writer itself?
With no beta releases and CPAN installations picking up the latest version automatically, this is close to ideal. People build the most recent version available and a build is usually run almost immediately after a release. One notable exception is mid-2011, where a number of builds ran with old releases. Let’s cross the two versions to see how the trends interact — this is release date against release date with no consideration for when the test was run.
There’s a lot going on here, but this kind of cross-version matrix catches any number of incompatibilities.
Generally, each new release of XML::Writer is being tested with its contemporary perl. However, as we found initially, even the current releases get a test with an old perl. People aren’t testing old XML::Writers with new perl releases except for a very noticable streak for 0.4. That’s an intentional, thorough test of the old release against recent perls that we saw in mid-2011.
It goes without saying that CPAN Testers Statistics makes great use of the complete set of test information across all modules.
)