Daniel S. Katz

I recently attended the hybrid SC21 conference in person, as one of the about 3000 in-person attendees, along with several thousand virtual attendees. (I don’t know the exact numbers, but in the past, an SC conference would have about 12000 attendees in-person.) In December, I’ll be attending the hybrid AGU Fall meeting remotely.

Event formats, including hybrid

As some may know, the first image of a black hole was announced April 10. This quickly led to a lot of different institutions explaining how they were involved (e.g., my own University of Illinois), as well as a bunch of software projects explaining how their software was used (e.g., Matplotlib).  Those of us concerned with open source research software and its sustainability are trying to raise the profile of such software in research.

A call for funding agencies to require grantees to report on the research software they use

The Journal of Open Source Software: Using Open Source Practices to Build Community and Sustainability

The Principles of Open Scholarly Infrastructure

JOSS’s Commitment to the Principles of Open Scholarly Infrastructure

The Principles of Fauxpen Scholarly Infrastructure

Presentation - Making FAIR Fair to the Researchers (Force11 Conference - Force2024)

FAIR Principles for Research Software (FAIR4RS Principles)

Making Biomedical Research Software FAIR: Actionable Step-by-step Guidelines with a User-support Tool

This post is cross-posted from Upstream: https://doi.org/10.54900/zks0k-pe034  I recently attended the FORCE2024 conference at UCLA. I’m a member of the board of directors of FORCE11, the parent organization for the conference, and the co-located FORCE11 Scholarly Communications Institute (FSCI), which I was not able to attend this year but have taught at in the past.

Research Software, Citation, and FAIR at FORCE2024

Recently (19-20 September 2016), a group convened at a workshop in San Diego under the auspices of FORCE11 (and with funding from the Helmsley Trust) to work on understanding the Scholarly Commons, a scholarly communications ecosystem designed for 21st century scholarship across all scholarly fields.  Since the Scholarly Commons doesn’t yet exist, the overall goal is to define and to bring it into existence.

The Scholarly Commons: Challenges in Philosophy and Governance

I’m happy to announce that I’ve been awarded an inaugural fellowship from the Better Scientific Software (BSSw) Project, along with my great colleagues, Jeff Carver, Ivo Jimenez, and Andrew Lumsdaine.

Better Scientific Software Fellowship

I recently led the writing of a report on software sustainability and high-energy physics: https://arxiv.org/abs/2010.05102. This report effectively has three groups of authors, with each group listed in alphabetic order: Katz – Stewart, the organizers of the workshop, leaders of the analysis and summarization, and lead authors of the report; Assamagan – Sexton-Kennedy, speakers in the workshop and authors of sections of the report;

Authorship Tiers

I am writing to point out an interesting experiment that’s started up in the ACM Transactions on Mathematical Software (TOMS) computational journal.  There’s an editorial by Mike Heroux that explains the process. In brief, TOMS has created a “Replicated Computational Results (RCR)” review process, which is really a designation saying that the computational results published in an article are replicable.

Interesting Replicable “Badge” for journal articles

Eric Lander, who co-chairs the President’s Council of Advisors on Science and Technology (PCAST) for the White House, gave a great talk at the 2015 National Math Festival in Washington, DC on April 16th, called “The Miracle Machine.”  The full talk is available from this page, specifically as a Word document.  Eric’s abstract was: Many governments are currently redirecting money for basic research into more applied areas.

Eric Lander’s argument for public funding of basic research

This blog post is intended as companion text for a talk I gave at the September 2018 NumFOCUS Project Forum in in New York, though I also hope it stands on its own. To address software sustainability, it is important first to understand how the term sustainability is used more generally.  It’s most often used in the context of ecology, often specifically in the relationship between humans and the planet.

Fundamentals of Software Sustainability

This is a third (and last) in a series of short (ok, the first and second were actually short, even if this one isn’t) blogs related to talks and discussions at the 10th RDA Plenary. Software Heritage   Software Heritage (as described in a recent white paper) is a really interesting and very ambitious project that started (or at least was announced) last summer.

Software Heritage and repository metadata: a software citation solution

Daniel S. Katz and Rajiv Ramnath, National Science Foundation {dkatz, rramnath}@nsf.gov   (A white paper for the Computational Science &amp;

Looking at Software Sustainability and Productivity Challenges from NSF

Based on some recent experiences, I want to try to redefine cyberinfrastructure. While I think Craig Stewart’s 2010 definition is still useful, new thinking might also have benefits. Let’s start with the elements of cyberinfrastructure. At the most basic level, these elements are simply people and things.

Redefining Cyberinfrastructure

This blog is based on part of a talk I gave in January 2017, and the thinking behind it, in turn, is based on my view of a series of recent talks and blogs, and how they might be fit together. The short summary is that general software reproducibly is hard at best, and may not be practical except in special cases.

Is software reproducibility possible and practical?

Once again, I’m writing in response to Titus Brown, who answered Is software a primary product of science? with “no” in his blog and on Twitter:  Note that Titus cleverly changed the question from asking about research to asking about science. Perhaps we can combine these and think about scientific research. So, what are primary products of scientific research?

Software can be a primary product of scientific research

While I was a program director at NSF, I led the SI2 (Software Infrastructure for Sustained Innovation) program that had been created by Manish Parashar and Abani Patra, then was led by Gabrielle Allen, before it was my turn to shape it and lead it for about four years, from 2012 to 2016. This program initially funded projects that developed and maintained software, as well as projects that planned and built community institutes around software.

The research community is moving towards the acceptance that data and software are both essential parts of understanding and reproducing many research (e.g., science, engineering, humanities) outcomes (e.g., papers, books, presentations) and are themselves valuable outputs that can capture and explain knowledge.  However, we don’t currently have a good set of practices to support reuse and sharing of data and software.

Data and Software Management Plans must be public and should be machine-readable

“Our Scholarly Recognition System Doesn’t Still Work” is the title of a panel at the Science of Team Science Conference that I’m co-organizing with Amy Brand (Digital Science), Melissa Haendel (Oregon Health &amp; Science University), and Holly J. Falk-Krzesinski (Elsevier)  If you are interested in this topic, you may want to consider attending the conference and the panel.

Our Scholarly Recognition System Doesn’t Still Work

(This is taken in part from another blog post.  Doug Thain asked a few of us to participate in a group blog on reproducibility, and this was my initial post on that blog.)  Given my two roles, a researcher and a funder, it’s clear to me that reproducibility in science is increasingly seen as a concern, at least a high level. And thus, making science more reproducible is a challenge that many people want to solve.

Mark Twain on Reproducibility, more or less

(by Daniel S. Katz and Tom Honeyman)  There are a number of challenges in rewarding scholars for their work in software, including the fact that both software itself and the idea of it being of scholarly value are relatively new, particularly given the centuries of experience we have with journals and the Humboldtian university model.

Software is a process, not a series of products: the failure of traditional metrics

I’m really excited to have been able to share some thoughts and stories through a recent RSE Stories podcast, and after being interviewed for this by Vanessa Sochat (which was a great experience; Vanessa is a good interviewer and a good editor), I realized that I left out one story I meant to tell, and decided to write it here.

Valuing people vs hardware

As I’ve been involved in the new FAIR for Research Software (FAIR4RS) activity (jointly run under FORCE11, RDA, and ReSA), I’ve been leading a subgroup looking at the FAIR (data) principles and thinking about how they can be applied to software. One of the things that has become clear to me is how fundamentally different software and data are, and the consequences this has on how software can be made FAIR.

Software and software metadata

Following Titus Brown’s blog post, A framework for thinking about Open Source Sustainability?, which follows Nadia Eghbal’s, An alternate ending to the tragedy of the commons, I’ve finally started reading Elinor Ostrom’s Governing the Commons. I urge you to read Nadia’s and Titus’ blog posts.

Following Ostrom-Eghbal-Brown into thinking about open source sustainability

I would like to have a well-functioning, easy-to-use, software citation system in place. In brief, I believe that doing so would increase the recognition of the role of software and its developers and maintainer in research, and would provide them credit in a manner that would be useful in the academic and related settings in which many of them work.

Compact identifiers for software: The last missing link in user-oriented software citation?

Measuring software impact is important, for a variety of reasons, and a variety of people, including software creators, their employers, their funders, potential users of the software, and academics studying software and science.  This topic has been studied by many, including some nice work by Thain, Tannenbaum, and Livny about 10 years ago.

Software metrics – what to count and how

In response to Doug Thain’s question: What currently-available tools do you recommend for enabling reproducible scientific computing?  Is there a tool that we ought to have, but do not? P.S. I am using “reproducibility” as an easy shorthand for re-usability, re-creation, verification, and related tasks that have already seen some discussion.  Please interpret the question broadly.

In response to Doug Thain’s question:  I think the most important thing we need is an electronic lab notebook that really allows us to go back and understand exactly what we did, repeat it, modify it, record it, etc.  If you accept this, it leads to a number of points: Why don’t more people (including me) do this now? What tool(s) should we use?

The Need for Notebooks

The latest Research Software Alliance (ReSA) newsletter includes the fact that: The 2018 European Commission report, Turning FAIR into Reality, concludes that FAIR digital objects (including software) need to be supported by metrics, incentives, skills and FAIR services that provide persistent identifiers, metadata specifications, stewardship and repositories, actionable policies and Output Management Plans.

Software and repositories in the context of FAIR

Since we often forget what reproducibility means, particularly in the context of repeatability and replicability, I think it is worth highlighting this slide, taken from a talk by Neil Chue Hong, “Open Software for Open Science,” Young Alliance Against Cancer, 22-23 May 2015, Copenhagen, http://dx.doi.org/10.6084/m9.figshare.1424440

 Note

Since the Reproducible Scientific Computing blog created and curated by Doug

Re****bility

The problem

I’ve been thinking about software catalogs for a while, mostly since the program I lead at NSF (Software Infrastructure for Sustained Innovation, SI

 2

, see the second half of http://nsf.gov/si2) has funded a fairly large number of projects, around 100.

Catalogs and Indices for Finding (Scientific) Software

I’ve been watching and participating in FAIR work for a while, first in terms of data, and more recently in research software, workflows, and machine learning models.

FAIR is not the end goal

(Please cite this post as https://doi.org/10.59350/rjt2m-2hy19)  A few things recently have made me think about how we support the core work of maintaining research software and supporting its users. I was interviewed by Georgia Iacovou for her report, “The state of the open source ecosystem &amp;

Supporting the core work in research software

After a number of general discussions in the research communication community, mostly focused on software citation, and then a few separate discussions with Anita Bandrowski and Martin Fenner, it’s become clear to me that we need something like a group (perhaps hierarchical) object identifier (GROUPID), which is somewhat different than a DOI, or at least different than how DOIs are commonly used today.

To better understand research communication, we need a GROUPID (group object identifier)

In a series of tweets (starting with one from @dimazest and including some from @turingfan, @Iza_Romanowska, and @CIRCA_StAndrews), it’s become apparent that we don’t have a standard way of providing attribution (e.g., citations, acknowledgements, dependencies) within software like we do in papers, where we have standard acknowledgement and reference sections, and for at least the reference section, a standard practice

How should we add citations inside software?

The problem

The following question was emailed to me by Jim Fowler: “I’m a professor at Ohio State and I sit on the committee which is drafting a new IP policy for faculty work. I’d like the new policy to support faculty contributions to open source software projects. I am very impressed with what you’ve done to recommend open source licensing on NSF funded projects;

Open Source Software and University Intellectual Property Policies

This viewpoint is published jointly on software.ac.uk, hpcnotes.com (personal blog), danielskatzblog.wordpress.com (personal blog) under a CC-BY licence. It was written by Neil Chue Hong (Software Sustainability Institute), Simon Hettrick (Software Sustainability Institute), Andrew Jones (NAG), and Daniel S. Katz (University of Chicago &amp;

The Price of Open-Source Software – A Joint Response

Over the holidays, I saw an interesting article on F1000Research Soergel DAW.

Thinking about Open Peer Reviews

This is an expansion of a comment on Titus Brown’s blog post “Please destroy this software after publication. kthxbye.” which talked about how much work should go into software that was used in a submitted paper. In the past, I’ve thought of software as having one of two different purposes: Some software is just written for a single research purpose – this can be quick and dirty, as long as it does the immediate job.

Scientific Software Quality versus Purpose?

While I’ve never been in an improv class, I really like the idea of saying “yes, and” as a way to move the activity forward. As others have also pointed out, this can work in business and research as well.

Collaboration: Working with Yes Anders rather than No Butters

Research software is an essential part of most research today, as I’ve written about before. One of the important models that allows people to develop and maintain it is grant funding: someone proposes building and/or maintaining software to a funder as an effort that aligns with the funder’s mission and programs.

Evaluating research software proposals

Make-or-buy decisions are common in business and research. They often involve an analysis of the pros and cons of creating something (e.g., a physical item, software, a service) vs. acquiring it from someone else. Factors that are considered can include cost, trust in suppliers, and current internal knowledge and skills as well as desired future knowledge and skills.

Make-or-buy decisions in research and research software

I’m very pleased to see that the Netherlands group working on “Software Management Plans: Towards national guidelines” has released a draft set of guidelines for community consultation (through 20 June 2022) and is organizing a workshop on 9 June to test and discuss the guidelines with use cases from the research community: https://www.eventbrite.com/e/software-that-makes-you-proud-using-smps-tickets-341888165707.

Progress on Software Management Plans in the Netherlands

I’m excited by this paper: “Recognizing the value of software: a software citation guide [version 2; peer review: 2 approved]” and want to provide some context here (also in a thread of tweets).  This is work by the Journals Task Force, which I lead with a lot of help from Shelley Stall. It’s part of the FORCE11 Software Citation Implementation Working Group, which I co-lead with Neil Chue Hong and Martin Fenner.

Progress in Software Citation in Scholarly Publishing

By Daniel S. Katz, Daina Bouquin, Neil Chue Hong    Identification of software is essential to a number of important issues, such as citation, provenance, and reproducibility. Here, we are focusing on issues related to citation. Identification can be thought of as a subset of naming. Some important questions are therefore: How do we name things? How do we know how things are named? And who gets to name things?

Towards Software Non-Creator-Instigated Identification (NCI) and Citation

This short blog post is really a pointer to another post that I co-wrote (with Sandra Gesing, Olivier Philippe, and Simon Hettrick), “Results from a US survey about Research Software Engineers,” as the initial post for the US Research Software Sustainability Institute (URSSI) blog.

Results from a US survey about Research Software Engineers

I’m happy to announce that I’ve started a project with rOpenSci under their recent award from the Helmsley Foundation. My work with rOpenSci will focus on sustainability of the project itself. Sustainability can be defined as having the resources to do the necessary work to continue and grow rOpenSci. This is one of the most difficult challenges for rOpenSci and for many other research software projects.

Software sustainability research with rOpenSci

This post is inspired by a post from C. Titus Brown, where Titus asked “What do you think about the term &amp; practice of ‘hardening software’?” and a few of us used comments to discuss the concept.

Hardening software vs making it sustainable

by Daniel S. Katz and Michelle Barker  Most researchers and policymakers support the idea of making research, and specifically research outputs, findable, accessible, interoperably, and reusable (FAIR). The concept of FAIR has been well-developed for research data, but this is not the case for all research products.

Generalizing FAIR

[Note: In December 2021, I decided to write down my experiences with the founding of US-RSE. Vanessa Sochat convinced me that it would make more sense for all interested US-RSE members to write a collective post, which we have now done as https://us-rse.org/2022-02-06-a-brief-history-of-usrse/. So much of this blog post overlaps that one, but some of it doesn’t, and I’m publishing this as a record of my recollection.

A kind-of brief (and personal) early history of US-RSE

The new international initiative, Initiative for Open Citations (I4OC), that started this week with the aim of promoting the unrestricted availability of scholarly citation data, is a wonderful idea.

Adding citations as part of the software publication process

The idea of a learning curve is fairly well-accepted, that one makes progress in gaining skills at some rate that can be “steep”, where it takes a long time to gain much proficiency, or “gentle”, where a fair amount of proficiency is gained relatively quickly. Here, I want to propose the parallel idea of a software development curve.

Introducing the Software Development Curve

I’ve previously written about the concept of workflows, sets of independent tasks connected by data dependencies, being expressed either as data, for example, a Pegasus DAG or CWL document, or as code, for example, a Python program written in Parsl.

Using workflows expressed as code and workflows expressed as data together

While preparing for a workshop that the URSSI project is hosting on software credit today, I started thinking again about a recent blog by Titus Brown, “Revisiting authorship, and JOSS software publications.” Titus says, “fundamentally, in order to nurture a diverse array of valuable scientific contributions, we need new models of publication with new models of authorship,” a statement with which I strongly agree, and in fact, part of the

Contribution vs Credit vs Authorship for software

by Daniel S. Katz and Kenton McHenry  We typically think of Research Software Engineers (RSEs) as working to support one or more researchers, either one-on-one or through a university’s centralised RSE group. We’ll call this the traditional RSE role, while being fully aware how ironic this phrase is. Simon Hettrick has used the following image, where “Here” indicates where the RSE fits between software engineering and research.

Super RSEs: Combining research and service in three dimensions of Research Software Engineering

The FAIR data principles, defined as “a set of guiding principles to make data

 F

indable,

 A

ccessible,

 I

nteroperable, and

 R

e-usable,“ came out of a meeting in Jan 2014 that “brought together 25 high level participants representing leading research infrastructures and policy institutes, publishers, semantic web specialists, innovators, computer scientists and experimental

FAIR is not fair enough

I’m currently the Steering Committee chair for the IEEE International Conference on eScience series.  (Note that I’m going to talk about IEEE in this blog, but the problem is really much more general;

Attempting conference changes: Why moving away from traditional publishers for proceedings is difficult

(a white paper for the 2019 Collegeville Workshop on Sustainable Scientific Software (CW3S19))

Data-driven software sustainability

When thinking about a group adding a new person, we often think that while the group will benefit from adding the new person in the long run, there is a cost to the group in the initial period.

How groups can immediately benefit when new members join

While at WSSSPE4 discussing possible WSSSPE mission statements, the question of what software sustainability is has been raised.

Defining Software Sustainability

(with contributions from Michelle Barker, Neil Chue Hong, Matthew Turk, Jeffrey Carver, Hannah Cohoon, and James Howison;

There’s no such thing as sustainable research software

This post is meant to summarize the

 “Our Scholarly Recognition System Doesn’t Still Work” p

anel, held at the Science of Team Science Conference (SciTS), Friday, June 5, 2015, Bethesda, Maryland, USA The panel organizers were Daniel S. Katz (U. of Chicago &amp; Argonne National Laboratory), Amy Brand (Digital Science), Melissa Haendel (Oregon Health &amp;

Summary of “Our Scholarly Recognition System Doesn’t Still Work” panel

About a month ago, I attended (thanks Sloan Foundation!) the Scientific Software Registry Collaboration Workshop, which had very interesting discussions. While there, I continued thinking about how software should be cited, focusing on open source. My thinking was certainly influenced by talking with Anita Bandrowski, Bryce Mecum, Shelley Stall, Katrina Fenlon, and Carly Robinson at the workshop.

Second thoughts on Proper Citation Guidance for Software Developers

The end of last week (26 March) marked the start of my fourth and final year as a rotator at NSF.  It’s been a really good experience so far, and I’m a bit sad to see the end coming. I’ve really enjoyed being able to work with a wonderful set of people in the Division of Advanced Cyberinfrastructure and across NSF.

One year to go at NSF

Having just returned from a short trip to the RDA Plenary 10, I’ve been thinking about a few things that will be the subject of a couple of hopefully short blogs in the near future.  This first one is about software identification, specifically as related to citation.

Key properties of software: agency and dynamicity

I’m writing this blog post in preparation for a contribution to the upcoming RDA Plenary 19 session “BoF: Sharing code that supports research articles: is it time to align goals and publisher policies?” at 0030 UTC on 23 June. I want to talk about how and why software (code) is shared.

Benefits of sharing code with articles

This is the second of a set of short blogs based on thoughts at the RDA Plenary 10.  (The first was on agency and dynamicity as key properties of software.)  In the Persistent ID (PID) interest group session at the RDA meeting, there was a talk by Patricia Cruse  about an organization ID project, which aims to create a new set of PIDs for organizations, followed by some discussion.

Group PIDs – a needed abstraction?

This post is a cross-post from the UK Software Sustainability Institute blog on 14 June 2017).

 By Daniel S. Katz, Assistant Director for Scientific Software and Applications at the National Center for Supercomputing Applications, University of Illinois Urbana-Champaign.


 This article is part of our series: A day in the software life, in which we will be asking researchers from all disciplines to discuss the tools that make

Workflow systems for science: programming in the large

This post is written in response to a recent article: “Beyond authorship: attribution, contribution, collaboration, and credit,”

 Learned Publishing

28(2), April 2015 (DOI: 10.1087/20150211). As a member of the CASRAI working group who “provided critical review of the [contributorship] taxonomy,” I am generally supportive of this idea of better and identifying contributions to scientific works.

Measuring Contributions: People vs Products?

(by Daniel S. Katz, Kenton McHenry, Jong S. Lee)  (in conjunction with a talk [slides, video] presented 28 September 2021 at SeptembRSE)  Over the past nine years there has been growing acceptance of the role of Research Software Engineers (RSEs) and the value they bring to projects.

Senior level RSE career paths (with an s)

Open Science is a great concept (and recently described really nicely by Titus Brown, who said in part, “scientific progress relies on the sharing of both scientific ideas and scientific methodology.”) This is a wonderful ideal, and it describes how a lot of scientists would like to act. But as we know, most scientists don’t actually act this way, or at least not as strongly as they could. Why not?

Clash of cultures: Why all science isn’t open science

This post begins with an idea from a response to an NIH request for information that I wrote with Kyle E. Niemeyer and Arfon M. Smith, discussing what I then thought were the two needed roles to make software citation work: creating/publishing and citing, which I referred to as push (creating the object to be cited) and pull (actually citing it.)  Upon further thought, I now believe that there are really three relevant steps, which can

Creation, publication, and citation: Issues in software citation versus paper and data citation

While at the Chan-Zuckerberg Initiative’s Open Science 2022 Annual Meeting a couple of weeks ago, I was struck by a comment from Demetris Cheatham about how she hadn’t known about the scientific open-source community until she was introduced to it fairly recently, even though she has a huge amount of experience with the larger open-source community. This was especially confounding when she shared that she realized upon

I was part of a panel in the WORKS workshop at the SC17 conference a couple of months ago, where I gave a short talk called “Expressing and Sharing Workflows” (slides). I’ve written this blog to put the slides into words and add some details I left out. First, what is a workflow? To me, it’s a set of tasks and dependencies between them.

Expressing workflows as code vs data

While on a call today for the ACM Emerging Interest Group on Reproducibility and Replicability, I realized that for computational reproducibility to become pervasive, we need to solve a scalability problem. Today, a lot of computational reproducibility is checked by hand, whether this is in terms of artifact evaluation in a conference or a paper, or with post-publication checks.

Scalable computational reproducibility

Inspired by talks at the International Workshop on Science Gateways a few weeks ago, in particular keynotes by David De Roure on Social Machines and Michelle Barker on the evolution of Science Gateways, I started thinking about what would happen if we considered all computational and data software in the form of science gateways. The definition of a science gateway has shifted a bit over time.

What if all research software was a science gateway?

As I’m getting ready for the first meeting of the Chan-Zuckerberg Initiative’s Essential Open Source Software for Science program, with grantees, advisors, funders, researchers, and industry partners, and with the goal “to identify shared needs and how funders might support these needs in the future”, I wonder how much software maintenance should cost, and how we might determine that answer.

How much should software maintenance cost?

I’ve been involved in Project CreDIT a bit, since I read an article on it in Nature: Allen, L. et al. 17 April 2014. Publishing: Credit where credit is due.

 Nature

508, 312–313, doi:10.1038/508312a and wrote to the authors, since it seemed to overlap some of the ideas I’d been thinking about called transitive credit: Katz, D.S. 2014.

Project CRediT and Contributorship Roles

In a recent meeting, there was some discussion about a potential workshop recommendation that funding agencies should offer relatively small grants that would just be used for maintaining existing software, but not doing any new development. I had very mixed feelings when I heard this, and decided I would write up my thoughts and see what others felt too.

Only Mostly Dead: Grants for Software Maintenance Only

Daniel S. Katz's blog

The end of last week (26 March) marked the start of my fourth and
final year as a [rotator](https://www.nsf.gov/careers/rotator/) at NSF.
 It's been a really good experience so far, and I'm a bit sad to see the
end coming.

I've really enjoyed being able to work with a wonderful set of people in
the [Division of Advanced
Cyberinfrastructure](http://www.nsf.gov/div/index.jsp?org=ACI) and
across NSF.  Everyone I've worked with is here because they want to
contribute to new gains in science, engineering, and education, and my
colleagues are willing to think creatively about how best to do this.

In particular, it's been a privilege to lead the Software Infrastructure
for Sustained Innovation (SI2) program and working group,
following Manish Parashar and Gabrielle Allen, working briefly with Rudi
Eigenmann, and currently with Rajiv Ramnath.  I'm proud of the [Software
Vision](http://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf12113)
we wrote, and the [plans for implementing it across
NSF](http://www.nsf.gov/funding/pgm_summ.jsp?pims_id=504817).  And I'm
happy to finally have [the process for funding full Software
Institutes](https://www.nsf.gov/publications/pub_summ.jsp?ods_key=nsf15553)
underway.

On the research side, I'm thankful to my collaborators, particularly
Mike Wilde and the U Chicago & Argonne Swift team, Shantenu Jha, Jon
Weissman, Omer Rana, Kyle Chard, Simon Caton, and Manish Parashar for
enabling me to usefully contribute to a variety of continuing projects
during my limited research time.

And I'm happy that I've been able to create a new research and
[community building activity](http://wssspe.researchcomputing.org.uk)
around sustainable software and its role in cyberinfrastructure,
including software citation and career paths, working with Gabrielle,
David Proctor, Neil Chue Hong, Manish, Matt Turk, Arfon Smith, Kaitlin
Thaney, and many others.

This really has been a wonderful opportunity, and I urge others in the
US to think about when the right time will be for you to spend a year or
two (or three or four) at NSF, contributing to the community and also
learning an incredible amount about both how things work on this side of
the curtain and also about what other exciting work is going on that you
probably didn't know.

Of course, things will continue, both in this next year and beyond.  SI2
is going well and the start of the Institutes will be a new exciting
phase.  In [WSSSPE](http://wssspe.researchcomputing.org.uk/), we'll
continue with a WSSSPE2.1 workshop at SciPy this summer and a larger
WSSSPE3 meeting in Boulder in September, and try to transition from a
set of people talking about good things to do to a set of people working
together to accomplish them.

Overall, I've worked both more and harder at NSF than I thought I would,
but it's been well worthwhile, and as I said at the beginning, I'm a bit
sad to see the end approaching in another 12 months.

## Disclaimer

Some work by the author was supported by the National Science Foundation
(NSF) while working at the Foundation; any opinion, finding, and
conclusions or recommendations expressed in this material are those of
the author and do not necessarily reflect the views of the NSF.


"Our Scholarly Recognition System Doesn't Still Work" is the title of a
panel at the [Science of Team Science
Conference](http://www.scienceofteamscience.org) that I'm co-organizing
with Amy Brand (Digital Science), Melissa Haendel (Oregon Health &
Science University), and Holly J. Falk-Krzesinski (Elsevier)

If you are interested in this topic, you may want to consider attending
the conference and the panel.  The conference is June 2-5, 2015, at NIH
in Bethesda, Maryland, just outside Washington DC.  There is no
registration fee. Our panel is June 5, at 3:15 pm in "Balcony A."

Our submitted abstract is:

With a historical focus on individual disciplinary achievements, the
scientific community has been slow to figure out how to adequately
recognize and reward accomplishments by individuals that occur across
disciplines and in the context of collaborative work. This is a serious
impediment to fostering effective team science.

This panel will discuss issues around scholarly recognition,
specifically authorship, contribution, credit, and attribution as
associated with the development of scholarly products. Current models of
shared authorship and attribution are an obstruction to scientific
collaboration. When there are multiple authors, we tend to rely on the
order in which names are listed to determine the most significant
contribution, when in fact there are no consistent name-ordering
practices from one ﬁeld to another. Yet who gets credit for research and
discovery has a tremendous impact on people's lives. It affects career
advancement and tenure, as well as the transparency and integrity of the
permanent research record. Even in fields such as economics in which the
author order is alphabetical and a supposition of relative contributions
has been removed, it has been shown that one is more likely to get
tenure or win a prestigious prize if your last name begins with a letter
earlier in the alphabet
(<http://pubs.aeaweb.org/doi/pdfplus/10.1257/089533006775526085>).

How we apportion credit for collaborative works today is highly
subjective, open to abuse, and often determined more by lab politics or
seniority than by effort or contribution. Junior researchers and those
making non-traditional research contributions such as data and code tend
to lose out most on deserved recognition. As interdisciplinary
collaboration and multi-authorship increase across all fields of
research, we clearly need a better system for representing collaborative
contribution to published works --- film credits are one alternative
model. If this initiative is ultimately successful, there will be fewer
barriers to team science, fewer author disputes, and fewer disincentives
for sharing data and code, for example, because those contributions will
be more reliably recognized. Hence these efforts could positively
influence both the cooperative culture of research, and academic
incentive structures more generally.

"We will need to find better ways to do team science and reward it if we
are to solve large overarching problems. Everybody on the team needs to
get the same big gaudy championship ring..." \[AG Gilman. Silver Spoons
and Other Personal Reflections. Annu. Rev. Pharmacol. Toxicol, 2012\]

This area is of interest to funding agencies such as NSF (for example,
see <http://www.nsf.gov/pubs/2014/nsf14059/nsf14059.jsp>) and NIH in the
US, publishers, university administrations, and scientific researchers,
and the science of science research community.  Much work has been going
on in this area, for example, a recent effort to develop an open
standard for tagging contributor roles in multi-author research
publications (see <http://credit.casrai.org/proposed-taxonomy/>).
Project CredIT ([projectcredit.net](http://projectcredit.net/)) formally
launched in 2014 to address the groundswell of interest among
researchers, funding agencies, academic institutions, and editors in
increasing the transparency of research contributions, and in more
fine-grained attribution and associated credit tracking. The taxonomy is
now being actively piloted.  Another example is the concept of
transitive credit, where similarly, all contributions to a product are
registered, but in this case, quantitatively rather than qualitatively.
Efforts in VIVO and eagle-i have aimed to relate a person to the things
they do and create in support of expertise finding and attribution. A
Force11 working group (<https://www.force11.org/group/attributionwg>)
has been created to bring some of these efforts together.

This panel will include relatively brief (5 min) statements from some of
the research projects and researchers in this area, followed by a
substantive discussion between the panelists and the audience.

Speakers:

1. Amy Brand (Digital Science) (will not be available, slides to be
 presented by Dan Katz) 
 Project CRediT: recording contributor roles, see
 <http://www.nature.com/news/publishing-credit-where-credit-is-due-1.15033>
 and <http://projectcredit.net> for more details.
2. Robin Champieux (Oregon Health & Science University) 
 Force11 Attribution Working Group: linking attribution research and
 implementation activities, see
 <https://www.force11.org/group/attributionwg> for more details.
3. Holly Falk-Krzesinski (Elsevier). Team science reward and
 recognition and publishers' role in clarifying attribution in a
 digital world. See [Mendeley Science of Team Science
 (SciTS)](http://www.mendeley.com/groups/3556001/science-of-team-science-scits/),
 specifically subgroups "Credit_Promotion and Tenure" and
 "Authorship_Publishing Issues" (available via desktop client) for
 more details.
4. Daniel S. Katz (University of Chicago & Argonne National
 Laboratory) 
 Transitive Credit: recording weighted credit for both contributors
 and resources, see <http://dx.doi.org/10.5334/jors.be> for more
 details.
5. Philippa Saunders (University of Edinburgh) 
 The Academy of Medical Sciences Team Science policy project:
 examining researchers' incentives and disincentives to participate
 in large collaborative projects, focusing on how such contributions
 can be better recognized in career-relevant decision making, see
 <http://www.acmedsci.ac.uk/policy/policy-projects/team-science/> for
 more details.

## Disclaimer

Some work by the author was supported by the National Science Foundation
(NSF) while working at the Foundation; any opinion, finding, and
conclusions or recommendations expressed in this material are those of
the author and do not necessarily reflect the views of the NSF.

## The problem

I've been thinking about software catalogs for a while, mostly since the
program I lead at NSF (Software Infrastructure for Sustained Innovation,
SI^2^, see the second half of <http://nsf.gov/si2>) has funded a fairly
large number of projects, around 100.  Many of these projects are
producing and supporting a piece of software, but some contribute to
multiple pieces of software, and in some cases, multiple projects
contribute to the same piece of software.  In addition, some of the
projects were funded long enough ago that they have ended or will end
soon.

The issue of catalogs or indices is important in multiple ways:

- As a funder, I would like to have a list of the products that we are
 supporting or have supported, tied to the projects that we have
 funded.
- As a user, I would like to know what software is likely to be around
 for a while, and NSF funding (particularly with the period of funding
 shown) would be a useful thing to know.
- As a developer, I want others to be able to easily find (and then both
 use and cite) my software.

## What happens in other areas?

In the publishing and web content worlds, this is somewhat similar to
the roles of a publisher/aggregator/distributor, a content consumer, and
a content creator.

Important artifacts are:

- The identifiers of each item of content that enable the items to be
 uniquely and consistently identified: e.g., DOI, ISBN, ISSN, URL
- The catalogs and indices that store content identifiers and related
 metadata (e.g., creator, publication date, format), and allow
 consumers to find them: e.g., publisher catalog, journal table of
 contents, curated pages of links, best-seller lists, Google, Google
 Scholar
- The services and tools that generate such catalogs and indices,
 whether public or private: e.g., Google Scholar, Mendeley, university
 profiles and knowledge systems

## What some government programs are doing

We (the SI2 program) have talked about this a lot, and what we decided
to do, partially as a compromise between effort and reward, is a google
sites page: <http://bit.ly/sw-ci>

NIH, as part of its Big Data to Knowledge (BD2K) initiative, held a
"[Software Discovery Index](http://softwarediscoveryindex.org)" workshop
in May 2014, with a [report](http://softwarediscoveryindex.org/report/)
available.

The Department of Energy's Office of Science's X-Stack program maintains
a [wiki](https://xstackwiki.modelado.org/Extreme_Scale_Software_Stack)
to track its current projects.

DARPA has an [open catalog](http://www.darpa.mil/opencatalog/), and
other DARPA programs may join this catalog. NASA's
[code.nasa.gov](http://code.nasa.gov/) is based on the same idea.

All of these have value, but none are really fully satisfying.  There
are issues with both the SI2 page and the X-Stack page with old projects
either needing to be manually removed or being forcibly removed (listing
is tied to funding, not to the ongoing value of the project).
 Centralized maintenance of these pages seems to be part of the problem.
 The DARPA and NASA catalogs link to software repos, but not DOIs, and
require metadata JSON files to be placed in a particular GitHub
repository, which is used to build the catalogs.

While not a government program,
[freshmeat/freecode](http://freecode.com/articles/whats-in-a-name) is an
interesting example of a catalog [failing](http://freecode.com/about)
due to low utilization and perhaps, impractical maintenance. \[pointer
from James Howison\]

## Proposed solution

Perhaps we can build an infrastructure that allows content creators and
users both to contribute?  Here's a set of ideas that might make this
work:

- Repositories provide a simple option to publish a release, [like
 one can do from GitHub and
 Zenodo](https://guides.github.com/activities/citable-code/), but in
 fewer steps and with fewer changes of servers. (Note: I assume that
 publication of software is a conscious decision that is made at
 particular times --- releases.  An alternate model that could also be
 explored is that publication happens by default as software is checked
 in to repositories.)
- The person who publishes a release fills in metadata about the release
 (as is done on the [Zenodo](https://zenodo.org) page now.)   These
 releases would then be assigned DOIs. And they would become catalog
 entries through the work of a crawler or future system like Google
 Scholar (let's call it OSC -- open software catalog -- for now).
  Additionally, they could also be crawled or curated for discipline
 specific purposes.
 - Generating metadata for repositories is a larger challenge than just
 for software. The idea of user-generated metadata, as suggested here
 and by
 [others](https://scholar.google.com/scholar?q=user-generated+metadata),
 is a potential answer.
 - In order for user-generated metadata to work, we as a community need
 to agree on the standards for such metadata (equivalent to [work
 going on for
 data](https://rd-alliance.org/groups/metadata-standards-directory-working-group.html)
 in the [Research Data Alliance](https://rd-alliance.org).)  Ideally,
 there would be a minimal set of required metadata, with optional
 extensions, perhaps even discipline-specific options.
 - A related question is how much of the needed metadata can be
 autogenerated?  For example, commits could be used to generate
 author lists, or at least a starting point for a person to use in
 defining the authors. (Note that doing this kind of autogeneration
 is useful in credit in general; see [Implementing Transitive Credit
 with JSON-LD](http://arxiv.org/abs/1407.5117) and [Project
 CRedIT](http://projectcredit.net/) as example of where it could be
 used.)
 - A [DOAP RDF file](https://projects.apache.org/create.html) could be
 used to store this information.
- The person who publishes a release can optionally indicate the funding
 source (with the funding end date), which is added to the metadata,
 and visible perhaps as a badge/flag, once the funder affirms that it
 is correct.  When the end date passes, the badge/flag would change to
 indicate prior funding.  If the projects gets new funding, a new
 badge/flag can be added.
- Quality metrics could be also generated, either by peers or by
 automated testing.
 - Users could give ratings and discuss successes and failures with the
 software (as is done in [NanoHub](https://nanohub.org) and App
 Stores).  Citations to the software from papers could also
 automatically be generated.
 - Quality metrics could be automatically generated from
 developer-provided test suites.  Note that this gets back to the
 question of standard practices as in the [DARPA open
 catalog](http://www.darpa.mil/opencatalog/) -- it would be better if
 this was directly supported within the repositories rather than by
 the catalogs).  This would also potentially address the issue of bit
 rot (somewhat related to [an idea from James
 Howison](http://dx.doi.org/10.6084/m9.figshare.1111632).)
- People looking for software to use or to further develop can search
 the catalog.  It's unclear how much this would happen vs. just talking
 to colleagues to see what they use, but perhaps these
 badges/flags/scores would make this more common.
- A related idea is that catalog could be partially automatically
 generated, and partially curated.  This idea builds on the idea of a
 software wikipedia to enable developers to know who is doing what and
 collaborate rather than compete (even unintentionally), as was
 suggested by one of the attendees of the [2015 SI2 PI
 workshop](http://cococubed.asu.edu/si2pimeeting2015/index.html).
 (Sorry I don't remember who suggested this.)

## Others discussing (or even working on) this

- Alice Allen: [Astronomy Source Code Library (ASCL)](http://ascl.net/),
 with manual entries and manual curation
- Geoffrey Bilder, Jennifer Lin, Cameron Neylon: [Principles for Open
 Scholarly
 Infrastructures](http://cameronneylon.net/blog/principles-for-open-scholarly-infrastructures/)
- Jason Duley: [Code.NASA.gov and Empowering the Open Source
 Community](http://open.nasa.gov/blog/2014/10/06/code-nasa-gov-and-empowering-the-open-source-community/)
- Martin Fenner: [Metrics for scientific
 software](http://blog.martinfenner.org/2015/02/19/metrics-for-scientific-software/)
- Paul Groth: [Trip
 report](https://thinklinks.wordpress.com/2015/02/02/trip-report-nsf-software-data-citation-workshop/)
 from [Supporting Scientific Discovery through Norms and Practices for
 Software and Data Citation and
 Attribution](https://softwaredatacitation.org/) workshop
- Arfon Smith: [JSON-LD for software discovery, reuse and
 credit](http://www.arfon.org/json-ld-for-software-discovery-reuse-and-credit)
- Jure Triglav: [Discovery of scientific
 software](http://juretriglav.si/discovery-of-scientific-software/)
 and [ScienceToolbox](http://www.sciencetoolbox.org)

(I'm happy to add more links here -- please email me)

## Acknowledgements

Thanks for Amy Friedlander, Martin Fenner, David Proctor, James Howison,
Chris Mattman, and Arfon Smith for useful comments on a earlier draft of
this blog.  The suggestions have been very help, and any errors are
solely mine.

## Disclaimer

Some work by the author was supported by the National Science Foundation
(NSF) while working at the Foundation; any opinion, finding, and
conclusions or recommendations expressed in this material are those of
the author and do not necessarily reflect the views of the NSF.

## Comments are welcome. {#comments-are-welcome.}

I've been involved in [Project CreDIT](http://credit.casrai.org) a bit,
since I read an article on it in Nature:

Allen, L. et al. 17 April 2014. Publishing: Credit where credit is due.
*Nature* 508, 312--313, doi:[10.1038/508312a](http://dx.doi.org/10.1038/508312a)

and wrote to the authors, since it seemed to overlap some of the ideas
I'd been thinking about called transitive credit:

Katz, D.S. 2014. Transitive Credit as a Means to Address Social and
Technological Concerns Stemming from Citation and Attribution of Digital
Products. *Journal of Open Research Software* 2(1):e20,
doi:[10.5334/jors.be](http://dx.doi.org/10.5334/jors.be)

This led to me joining the [project working
group](http://credit.casrai.org/working-group/), participating in a
series of phone calls to discuss the specific [taxonomy of
contributorship roles](http://credit.casrai.org/proposed-taxonomy/), and
attending a
[workshop](http://www.eventbrite.ca/e/project-credit-workshop-tickets-10314211083) that
was intended to promote discussion about and trials of the roles.

The thing that makes this interesting to me is that these roles ideally
could be used to describe who contributed and how they contributed to
any scholarly product in any domain, for example, papers, software,
data, books, physical objects, etc.  As you might imagine, it's not
immediately clear if
the [contributorship taxonomy](http://credit.casrai.org/proposed-taxonomy/)
can be both general enough to cover all products and also useful enough
to make people want to use it.

While thinking about this at the workshop, and in some discussions
afterwards, I realized that there is a time element that probably needs
to be considered.  For a paper, at least, I think about the work that is
done (and needs to be recognized in the contributorship model) as having
three phases:

1.  **planning**, including: conceptualization, methodology, resources,
    funding acquisition, etc.
2.  **doing**, including investigation, formal analysis, software
    (building code for research), writing (original), validation
    (authors ensure the answers are right), etc.
3.  **reporting**, including data curation, software (packaging and
    archiving for reproducibility), writing (both original and review &
    editing), validation (others outside the team ensure the answers are
    right), etc.

Some roles seem to occur in multiple phases, such as software, writing,
and validation, which makes me think that these are yet well-enough
defined.  And this also make me quite strongly think that having
software listed under data curation is wrong.  This item is too
data-centric.

For a project that is aimed at creating software or data, but not at
using it to gain new knowledge immediately, most of the roles still make
sense, but the definition may be a bit off.  For example, what does
investigation mean?  Perhaps this is planning the software, determining
what algorithms to use, etc.?

I can imagine a set of tables (one for each kind of product, e.g.,
paper, software, data, maybe different for different domains?) would be
useful here, where each table has the set of roles as rows, and the
phases as columns, and Xs in the cells where that role fits in that
phase.  More than one X in a row might mean that role isn't sufficiently
well described yet.  Different Xs in different tables might also mean
the roles aren't correctly described yet.

Finally, the workshop included some discussion about how these roles
overlapped authorship, and the answer I gave, which seemed somewhat well
received, is that they are separate.  Contributorship is intended to be
a set of observed properties (i.e., researcher X performed role Y during
the project), while authorship is a set of cultural values that differ
across communities (e.g., should the person who acquires funding for a
project be an author?)

Other interesting questions are:

- Does this taxonomy apply across all scholarly domains? (It was
  originally conceived in a biomedical context.)  What about domains
  such as architecture?  Or art?
- Can this taxonomy be used all types of scholarly products?  Such as
  for a survey paper? Or for a research project that produces physical
  objects?
- Can the amount of role a researcher fulfills be measured?  Perhaps as
  lead & participant, major & minor, or numerically?
- How can this work be combined with the idea of transitive credit?


Over the holidays, I saw an interesting article on F1000Research

[Soergel DAW. Rampant software errors undermine scientific results \[v1;
ref status: approved with reservations
2, <http://f1000r.es/4w2>\]]{.article-title-and-info
.in-popup} *F1000Research*2014, **3**:303
(doi: [10.12688/f1000research.5930.1](http://dx.doi.org/10.12688/f1000research.5930.1){target="_blank"})

and felt inspired to comment on it.  The editors asked if I would use my
comment as the basis for a review, which I agree to, leading to my first
open peer review:

[Katz D. Referee Report For: Rampant software errors undermine
scientific results \[v1; ref status: approved with reservations 2,
<http://f1000r.es/4w2>\]]{.article-title-and-info .in-popup}
*F1000Research* 2014, **3**:303 (doi:
[10.5256/f1000research.6338.r7096](http://dx.doi.org/10.5256/f1000research.6338.r7096){target="_blank"})

This made me think about the peer review process, and what specifically
was different about open peer reviews (where the reviewer is known) vs
traditional closed peer reviews (where the review is anonymous.)

I mentioned this in a tweet, and someone pointed me to

[Aleksic J, Alexa A, Attwood TK *et al.* The Open Science Peer Review
Oath \[v1; ref status: indexed,
<http://f1000r.es/4ou>\]]{.article-title-and-info .in-popup}
*F1000Research* 2014, **3**:271 (doi:
[10.12688/f1000research.5686.1](http://dx.doi.org/10.12688/f1000research.5686.1){target="_blank"})

which led me think about this even more, leading the following, which is
also posted as a comment on the Open Science Peer Review Oath paper.
 Comments are welcome.

::: {.commenter-role style="padding-left:30px;"}
Reader Comment 22 Dec 2014
:::

::: {style="padding-left:30px;"}
[Daniel S. Katz]{.name}, [University of Chicago,
USA]{.commenter-affiliation}
:::

::: {style="padding-left:30px;"}
:::

::: {.comment-content .article-comment-list style="padding-left:30px;"}
::: {#full-comment-1141 .padding-bottom}
​The suggested oath, broken down by principles and items, is quite
interesting, with some of the principles having clear value and little
overhead, while others that have potential value also add large amounts
of possible overhead.  I suggest that these be broken down into two
types of open reviews.  Specifically:

**Passive review**: 
Follows all of Principles 1 through 3, and elements x -- xii of
Principle 4.  Here, the idea of an open review basically is a review
that is signed to make sure the process has appropriate integrity.

**Active review**: 
Includes all elements of Passive review, as well as items xiii and xiv
of Principle 4:
:::

::: {.padding-bottom}
:::

::: {.padding-bottom}
- xiii) I will check that the data, software code and digital object
 identifiers are correct, and the models presented are archived,
 referenced, and accessible
- xiv) I will comment on how well you have achieved transparency, in
 terms of materials and methodology, data and code access, versioning,
 algorithms, software parameters and standards, so that your
 experiments can be repeated independently

The idea of an open review here seems to be that, in addition to
ensuring integrity in the review process, there is also extra work being
done beyond a standard review, and the person who does such work should
be credited for doing so.

I am uncertain about where elements xv and xvi, as written:
:::

::: {.padding-bottom}
:::

::: {.padding-bottom}
- xv) I will encourage deposition with long-term unrestricted access to
 the data that underpin the published concept, towards transparency and
 re-use;
- xvi) I will encourage central long-term unrestricted access to any
 software code and support documentation that underpin the published
 concept, both for reproducibility of results and software availability

would fit.  These are neither active nor passive, and as written, they
don't match the review function, but are even more active, more a
collaboration than a review.  I suggest that they be rephrased as:
:::

::: {.padding-bottom}
:::

::: {.padding-bottom}
- xv) I will check that the data that underpin the published concept are
 made available in a manner that provides long-term unrestricted
 access, towards transparency and re-use;
- xvi) I will check that any software code and support documentation
 that underpin the published concept are made available in a manner
 that provides long-term unrestricted access, both for reproducibility
 of results and software availability

so that they could be part of an Active review.

Of course, this specific remedy is just a suggestion, but I the overall
point I want to make is that the added work to be done by the reviewer
beyond what is now standard needs to be explicitly considered in both
the oath itself as well as the description of the oath.
:::

[[**Competing Interests:** [none]{.c-i-d}]{.disclosures
.competing-interests-display .padding-top}]{.full-comment}
:::