Developer Turnover in Open-Source Software

Matthieu Foucault, Marc Palyart, Xavier Blanc, Gail C. Murphy, and Jean-Rémy Falleri

Replication Package

This replication package allows the reproduction of the result presented in our FSE'15 paper, as well as the additional results presented below. We describe here how to reproduce the actual results, and how our extraction and analysis code can be reused.

Dataset

The file dataset.json contains the information extracted using the process described in the Section 5 of the paper. This file has the following structure:

{
	"repo_url_1" : {
		"authors" : {
			"johnDoe" : "John Doe" // all occurrences of "johnDoe" will be replaced by "John Doe"
		},
		"bug-fix-commits" : [
		"ab12321", // The sha id of bug-fixing commits
		"abc1234"
		],
		"cloc-commit-id": "519bef4f3", // The S_0 release (called 'cloc-commit-id' here because we extract the number of lines of code for this commit)
		"file-filter": "^src.*.js", // only files matching this regular expression are considered
        "modules": [ // each regular expression represents one module. If this attribute is not present, files are considered as modules
	        "^src/a/.*",
	        "^src/b/.*",
        ],
        "R_last":"9d53e5a3", // we do not analyse commits newer than this one (used for RQ1)
        "R_first":"c9c176a", // we do not analyse commits older than this one (used for RQ1)
        "releases": [ // the three releases used when considering releases for period length (see additional results)
        	"519bef4",
        	"8bae2a5",
        	"2a9579a"
        ],
	},
	"repo_url_2" : {
	...
	}
}
						
Note that in the case of JQuery, the URL of the repository refers to a fork of the original repository, due to the fact that in the JQuery repository, older maintenance branches are regularly deleted.

Replication (without dependency installation)

If your goal is to reproduce the same results presented in the paper, this is the procedure to follow. Note that some results may differ slightly from the ones in the accepted version of the paper, as we fixed defects in the analysis code (see additional results below)

Download and extract the VirtualBox VM image available here.

Data Analysis

Developers Contributions were extracted and exported in a set of CSV files, available in the csv-data.zip archive. This archive contains the following files:

  • dev_m_months.csv contains the monthly contributions of each developer to each module, prior to the S0 release of each project. The column 'custom_id' contains the index of the month, i.e. the lines with a 'custom_id' of '1' contains the contributions performed during the 1st month prior to S0, and so on.
  • dev_m_releases.csv contains the same contributions, for the two releases prior to S0. The releaseDate column should be used to determine whether contributions are from P1 or P2.
  • mod_m.csv contains metrics for each module: the number of bugfixes and the number of lines of code of the module at S0.
  • dev_m_after_S0.csv contains the monthly contributions performed after S0, which are used to compare the leavers sets obtained with limited and full visibility.
  • devdates.csv contains the commit dates of each developer, used to study turnover at the project level (RQ1)

These data were analysed using the R programming language. The various scripts used to produce the results are available in the scripts folder.

Additional Results

Period Selection

The text below contains several updates that could be partially included in the final version of the paper.

To choose a suitable size for the periods P1 and P2, we measured the impact of these periods on the sets of turnover actors (i.e. internal and external leavers and newcomers). The length of the periods P 1 and P 2 may impact the resulting sets of actors, especially if the periods are too short, in which case we may consider as newcomers or leavers developers who stopped contributing to the project for a while before starting again. To assess the impact of this choice we have tested four configurations for the lengths of the periods: one release-based configuration where S0 , S−1 and S−2 are three following releases of the project, and three time-based configurations where P1 and P2 both last for 1, 3 and 6 months.

The fact of using | P1 | = | P2 | limit our vision in the past. This may for example lead to consider developers as newcomers because they were inactive for some time but we do not see their old contributions. However, if we looked at the whole history of the project, to check whether developers are newcomers or leavers, we may consider as stayers developers who did not contribute to the project for several years. To quantify the impact of this limited visibility we decided to compare the sets of actors obtained when | P1 | = | P2| (limited visibility) and when | P1 | expands to the start of our observation period (full visibility) hence with S−2 = beginning of project’s history.

To decide which period size is suitable for our analyses, we chose to measure the similarity between sets of turnover actors computed with limited and full visibility, using the Sorensen-Dice quotient of similarity (which is equal to 1 when two sets are identical, and 0 when they are disjoint) [9]. We chose the first period size where the median quotient of similarity is, for all projects and actors sets, above a threshold of 0.75. The figure below presents the distribution of Dice coefficients obtained for each project. The release period configuration is not stable as the length of time between two releases depends on too many factors, and ranges from very different values. The monthly period configurations (either 1, 3 or 6 months) are much more adequate in our context. The similarity between sets of developers when using the periods of one or three months were relatively low in some projects, especially for internal turnover. Results obtained with the six months periods are much less sensitive to the limited visibility, as in all the projects the cases where there are large differences in the sets of developers represent a minority of modules.

Therefore, we chose to use the 6 months period for the remainder of our analysis.

Sorensen-Dice quotients of similarity between sets of turnover actors obtained with limited and full visibility.

RQ1: Turnover at the Project Level

Evolution of the number of developers

Evolution of the ratio of turnover actors over the total number of developers

RQ2: Patterns of Contribution

Erratum: We found and fixed several defects in the code extracting the developers contributions which impacts the patterns observed on Angular.JS. Most of the contributions are performed by stayers and external leavers. The contributions performed by external leavers are in mostly due to a single developer, a major contributor who did not contribute to the source code in the six months prior to the release of Angular.JS 1.0.0.

RQ3: Developer Turnover and Software Module Quality

Correlation between Bug density and developers activity
(|P| = one release)

Correlation between Bug density and developers activity
(|P| = one month)

Correlation between Bug density and developers activity
(|P| = three months)

Correlation between Bug density and developers activity
(|P| = six months)