4 Guide for Editors

4.1 Editor-in-Chief

As for rOpenSci’s current peer-review system, packages are submitted directly to the ropensci/software-review repository on GitHub, with submissions handled initially by the rotating Editor-in-Chief (EiC). All statistical software submissions should begin with a pre-submission enquiry, although direct submissions may be accepted at the EiC’s discretion. The EiC should perform the following duties on initial pre-submission enquiries of statistical software:

Check that the appropriate submission template has been used, and ask authors to edit and update data if not.
Ask authors to respond to any comments issued by the ropensci-review-bot about template format and content.
Call @ropensci-review-bot check srr to generate a summary of compliance with our “srr” (Software Review Roclets) system for documenting standards compliance.
Confirm from the report generated by check srr that the software complies with at least half of all applicable standards,
Perform other EiC responsibilities described in our main Dev Guide.

Full submissions generate a detailed report from our ropensci-review-bot, as described below. Examples can be seen by looking at any recent submissions. The EiC need only purvey the summary checks within the initial section of the report to confirm that all the submission passes all checks, and that the report concludes with a statement that,

This package may be submitted

In response to that statement, the sole tasks for an EiC prior to delegating a handling editor are to check the following single item:

The categories nominated by the submitting authors are appropriate for this package

And to choose one the following two items:

The package does not fit within any additional categories of statistical software.
The package could potentially be described by the following additional categories of statistical software:
- <list categories here>

Additional effort by the EiC will only be required in “edge cases” where a package may be unable to pass one of the checks, as in this sample automated check, which concludes with the statement that,

All failing checks above must be addressed prior to proceeding

In these cases, submitting authors must explain why these checks may fail, and the EiC must then determine whether these failures are acceptable. Such cases ought nevertheless be rare, and it may be expected in the majority of cases that the sole tasks of the EiC are to confirm a positive bot response, to complete the two checklist items given above, and to allocate a handling editor. This latter step is done by calling @ropensci-review-bot assign <name> as editor.

4.2 Handling Editor

The Handling Editor should use the summary report generated by the opening of the issue to perform an initial assessment and to guide assignment of reviewers. The contents of these automated checks are described at the end of this chapter. The EiC need only consider the initial summary checklist, but handling editors should consider all details contained within the automated report. General procedures for Handling Editors, including explanations of relevant ropensci-review-bot commands, are described in the Editor’s Chapter of the general Dev Guide. This section describes specific editorial requirements for statistical submissions.

4.2.1 Handling Editor Checklist

Having considered the automated package report, and addressed all issues raised within that report, Handling Editors should check the following items before assigning reviewers.

Any issues raised during initial processing by EiC have been resolved (or there were none).
Issues raised in the goodpractice checks, statistical anomalies, and other details of the report do not appear to be major problems and can be directed to reviewers for further scrutiny.
Either (i) this package is aiming for a bronze badge, or, (ii) for packages aiming for silver or gold badges, the authors have clarified which of the four aspects listed in the “Guide for Authors” section on silver badges they intend to fulfil.

Issues flagged by the handling editor may require iteration with submitting authors. As stated in the Dev Guide:

If authors believe changes might take time, apply the holding label to the submission.
If the package raises a new issue for rOpenSci policy, start a conversation in Slack or open a discussion on the rOpenSci forum to discuss it with other editors (example of policy discussion).

Once all items have been checked, Handling Editors may proceed to find and assign two reviewers. As for general software reviews, start by commenting @ropensci-review-bot seeking reviewers. A email template to invite prospective reviewers of statistical packages is in the appendix. Once reviewers have agreed, use the command @ropensci-review-bot add <@GITHUB_USERNAME> to reviewers, and generally follow the procedure given in the Dev Guide. Handling editors may provide guidance to reviewers on issues that they think may be worth looking into based on their initial review of the package and automated checks.

[Note that the Dev Guide is rapidly iterating as new capabilities are added to the review bot, and you may wish to refer to the draft version at <devdevguide.netlify.app/>.]

4.2.2 Re-generating package check results

The handling editor may update the initial package check results at any time with the following command:

@ropensci-review-bot check package

This is likely to be necessary following each review, to ensure any issues identified from the initial checks have been satisfactorily addressed.

4.2.3 Disagreements on Badge Grades

Handling editors are responsible for resolving any disagreements between authors’ stated or desired grade of badge and reviewers’ recommendations. Click here to jump to the corresponding recommendations for reviews. The views of reviewers should generally be prioritized in such cases. Grades as declared by authors are contained in the opening comment of the issue. These may be extracted by calling:

@ropensci-review-bot stats grade

Editors may modify these grades by editing the opening comment, and changing the value of the "statsgrade" variable.

4.2.4 Approval

After having completed a checklist and ensuring agreement on badge grade, the handling editor may approve a submission with the following command:

@ropensci-review-bot approve

The bot will identify that this is a statistical software issue, extract the appropriate grade, and attach a corresponding badge which will also label the latest version of our statistical standards.

4.3 Automated Checks

Upon initial submission, the ropensci-review-bot performs a suite of tests and checks, and will upload a report into the GitHub issue thread. This report is the primary source of information used to inform initial editorial decisions. All editors must familiarise themselves with the structure and contents of these automated reports, an example of which can be seen by clicking here. A similar report can be reproduced locally by running the code within the following sub-section.

Package report code here (click to see).

library (srr)
library (pkgcheck)
path <- srr_stats_pkg_skeleton ()
srr_stats_roxygen (category = "regression",
                   filename = file.path (path, "R", "srr-standards.R"))
check <- pkgcheck (path)
md <- checks_to_markdown (check, render = TRUE)

The crucial section for statistical packages is the first, which describes checks conducted by the srr (Software Review Roclets) package. This check confirms that all statistical standards have been documented within the code, and all packages must pass this check. The report linked to in that section is primarily intended to aid reviewers, and may be ignored by handling editors.

The second section describes the “Statistical Properties” of the package being submitted, and should be considered by handling editors. In particular, this section contains information which identifies any statistically noteworthy properties of the package. The example report illustrates how this report immediately identifies that the package has very little code, very few functions, and very few tests. Handling editors should consider these statistical details, and particularly any noteworthy aspects (defined by default as lying within upper or lower fifth percentiles in comparison with all current CRAN packages). Any aspects which seem concerning should be explicitly raised with submitting authors prior to proceeding. The measures currently considered include various metrics for:

Size of code base, both overall and in sub-directories
Numbers of files in various sub-directories
Numbers of functions
Numbers of documentation lines per function
Numbers of parameters per function
Numbers of blank lines

A final metric, fn_call_network_size, quantifies the number of inter-relationships between different functions. In R directories, these are function calls, while relationships may be more complex within src or inst directories. Small network sizes indicate packages which either construct few objects (functions), or in which internal objects have no direct relationships.

The third and final section of the automated report contains details of goodpractice checks including:

Code coverage estimates for each file (from the covr package).
Code style reports from the lintr package.
Cyclomatic complexity reports from the cyclocomp package.
Any errors, warnings, or notes raised by running R CMD check (from the rcmdcheck package.

Any aspects of these goodpractice reports which do not pass the initial checklist (such as warnings or errors from R CMD check, or test coverage < 75%) should be clarified with authors prior to proceeding with review.

Finally, the initial Statistical Description includes details of computer languages used in a package, and should be used to ensure reviewers have appropriate experience and abilities with the language(s) in which a package is written.