Policy

Estimated time: 24 minutes

Overview

Questions

• What is an official USGS software information product?
• When am I required to release my software as an official USGS software information product?
• When may I release my software as an official USGS software information product?

Objectives

• Identify the difference between a software project and an official USGS software information product.
• Explain requirements for releasing software as an official USGS software information product.
• Identify the policy hierarchy relationship among federal, agency, and USGS authorities.

Evolving Policy

Information conveyed in this episode reflects policy as it exists, or as it is anticipated to exist, at the time the lesson is initially made generally available within the USGS.

A best effort will be made to keep this policy information up-to-date moving forward; however, current published policy will always supersede what is found in this episode.

Verbal Emphasis

When speaking, emphasize software proJECT versus software information proDUCT. It is important these two words (project and product) are not confused as interchangeable. Also, note that a software information product is sometimes called a USGS code release, but that is not its formal name.

Source Code | Definition & Example

Computer commands written in a computer programming language that are meant to be read by people. As such, source code is a higher-level representation of computer commands and, therefore, must be assembled, interpreted, or compiled before a computer can execute it as a program.

Example

CPP

// file: hello.cpp
#include <stdio.h>

int main() {
  printf("Hello world!\n");
  return 0;
}

The above is an example of a file called “hello.cpp” that contains source code written in the C++ programming language. While the source code is relatively easily understood by a human, a computer is not able to execute this file directly.

BASH

$ ./hello.cpp
./hello.cpp: line 3: syntax error near unexpected token `('
./hello.cpp: line 3: `int main() {'

Instead this file must be compiled to an executable using a command similar to the following:

BASH

$ gcc -o hello hello.cpp

The resulting file, “hello”, contains binary machine code that can be executed by the computer.

BASH

$ ./hello
Hello world!

While C++ is an explicitly compiled language, other languages are more sublte and may leverage just-in-time compilation or interpretation of the source code. In these more subtle languages there may not be an explicit compilation command. Only the source code file exists and it is quietly compiled or interpreted behind the scenes upon execution.

Examples of these more subtle languages include Python or Shell scripts.

Source code developed by- or on behalf of- the U.S. Geological Survey that is- or intends to be- publicly accessible must be stored within a Git repository on the USGS Git Hosting Platform. This Git repository may have multiple branches, tags, and commits. There may exist issue trackers, build artifacts, milestones etc. Taken together, the activities and artifacts related to the prior, ongoing, or upcoming development activities of source code are considered a “software project”.

Official Software Information Product

---

When a software project reaches some level of maturity (e.g., results are used to support a published manuscript), it must be released as an “official USGS software information product”. While software projects may not be cited by other official USGS information product types (e.g., data releases, journal articles, etc.), official USGS software information products are citable. The desire to cite a software project is one example requiring the author to release the project as an official USGS software information product; however local policies (e.g., science center or equivalent organizational unit) may also define additional criteria requiring the author to release the project as an official USGS software information product.

An official USGS software information product reflects a point-in-time snapshot of a software project’s source code and relevant artifacts. This snapshot must be reviewed and receive appropriate approval to be made public. This snapshot is typically created using a Git tag in the repository and an associated GitLab release.

Open Source Software Project Development

---

A software project may be developed publicly as an open-source software project given the project complies with all governing policies. Subject to limited exceptions, current policy requires a software project must be made open-source when specific criteria are met, for example:

1. The project, or results thereof, are deemed sufficient to be used by the current or future research project(s)
2. A project that was contracted through a service contract vehicle is accepted by the federal contracting authority to satisfy contract requirements
3. The source code in the project is no longer considered truly exploratory or disposable in nature
4. The library or application produced by the software project is used by USGS or other federal staff on a regular, recurring basis
5. The library or application produces actionable information at scales and timeframes relevant to decision makers

When developing an open-source project, all contributions to the project must receive, at minimum, an administrative security review before the contribution is integrated into the project repository. Depending on the project, this review provides an opportunity to complete other types of review as well, e.g., technical code review.

Branching vs Forking Workflows

In this course we describe a branching workflow. This workflow is simpler for individual developers to understand when getting started with Git. However a forking workflow may be better suited for open source project development.

Current policy requires all contributions to open source projects be reviewed before they are integrated with the public project. Since all branches in the public project are themselves public, there is no way to use a branching workflow and comply with current policy. Under a branching workflow, the author must determine a method for sharing and reviewing their code prior to pushing their changes to GitLab; this may be fragile or lead to unversioned changes.

Conversely, a forking workflow enables reviews to occur by way of merge requests from the internal fork repository location to the public upstream repository location prior to integrating said contributions with the public project repository. In this way open source development may continue collaboratively while adhering to current policy requirements.

Governing policies (see below) determine the requirements for both open development practices and release of official USGS software information products, which are determined at each of Federal, Agency (Department of the Interior), Bureau (U.S. Geological Survey), and local (e.g., science center or equivalent organizational unit) levels.

Federal

Agency

Bureau

Local

1. Federal Source Code Policy: Achieving Efficiency, Transparency, and Innovation through Reusable and Open Source Software
2. 44 U.S.C. §3301. Definition of records

1. Open Source Software Policy

1. Survey Manual 500.2 - Policy on Work for Non-Federal Agencies
2. Survey Manual 500.3 - Policy on Work for Other Federal Agencies
3. Survey Manual 502.5 - Fundamental Science Practices: Safeguarding Unpublished USGS Scientific Information and Associated Materials
4. Survey Manual 1100.4 - Use of Outside Publications, Including Abstracts
5. Instructional Memorandum OSQI 2019-01 - Review and Approval of Scientific Software for Release

Mission areas, science centers, and other organizational units within the U.S. Geological Survey may also define their own policies. Check with your supervisor for guidance on what might be applicable from local policies.

In general, the policies are structured in a hierarchy such that higher level policy (e.g., federal policy) provides generalized guidance and lower level policy provides increasing specificity and clarity. Lower level policy may not supersede or conflict with higher level policy.

Challenge: Identifying relevant policies

Select from which source(s) there exist policies governing the release of official USGS software information products.

A. Federal

B. Departmental

C. Bureau

D. Local

Show me the solution

Policies are known to exist for each of (A), (B), and (C) sources. Science Centers and offices may implement additional policies with which you must comply.

Requirements

---

The following are required to release a software project as an official USGS software information product.

1. Proper license, disclaimer, and metadata (code.json file)
2. Appropriate review(s) and approval as defined by current policy
3. An approved Information Product Data System (IPDS) record
4. A Git tag within the project corresponding to the official USGS software information product and a GitLab release corresponding to the Git tag
5. A digital object identifier (DOI)

Other episodes in this lesson detail procedures to satisfy each of the preceding requirements.

Key Points

• Software may be publicly accessible as an open source software project and/or as an official USGS software information product.

• While both a project and product may be public, only the official USGS software information product is citable by other publications.

• Governing policies cascade from Federal to local levels. Check with your supervisor to ensure compliance with all local policies.