Artifact Evaluation

ARTIFACT EVALUATION

New to FCCM this year is the opportunity for authors to optionally participate in an artifact evaluation process.  Artifacts are digital objects that were created by the authors as part of the research or experiments performed with the submitted work.  Examples of artifacts are:

  • Software: Source code, scripts, Makefiles, container images (like Docker files), etc.
  • Hardware: Verilog, VHDL, schematics, CAD tools, flows, etc.
  • Data: Spreadsheets, databases, binary files, design sets, etc.

The goal of submitting artifacts promotes the availability and reproducibility of the experimental results and data such that other researchers can repeat experiments and replicate results with less effort. 

SUBMISSION REQUIREMENTS

If the authors would like to participate in artifact evaluation and are willing to prepare and document artifacts, they need to fill out and submit a copy of the Artifact Form alongside their paper when submissions are due. 

What is the Artifact Form?

The Artifact Form is a form used to collect information necessary for artifact evaluation.  The form allows the authors to describe the presence or absence of artifacts, and their type (software, hardware, or data) that supports the research presented in the paper.

Do I need to open-source my software in order to complete the Artifact Form?

No. You are not asked to make any changes to your computing environment or design process in order to complete the form. The form is meant to describe the computing environment in which you produced your results and any artifacts you wish to share. Any author-created software does not need to be open source unless you wish to be eligible for an artifact review badge.

REVIEW PROCESS

Who will review my artifact form?

If your submission is accepted as a full paper for publication at FCCM 2023, then the Artifact form will be reviewed by the Artifact Evaluation Chairs. The Artifact Evaluation Committee (AEC) will review the information provided and will verify artifacts are indeed available at the URLs provided. They will also help authors improve their forms, in a double-open arrangement. If authors select this option, their paper may be evaluated for artifact review badges.

How will the review of artifacts interact with the double-blind review process?

Artifact review will not take place until after decisions on papers have been made. Reviewers will not have access to the Artifact Form. Authors should not include links to their artifacts/repositories in their submitted paper. The paper review process is double-blind. The artifact review process is not.

IMPACT OF ARTIFACT EVALUATION

What’s the impact of an Artifact Form on scientific reproducibility?

An artifact-evaluation effort can increase the trustworthiness of computational results. It can be particularly effective in the case of results obtained using specialized computing platforms, not available to other researchers. Leadership computing platforms, novel testbeds, and experimental computing environments are of keen interest to the FPGA community. Access to these systems is typically limited, however. Thus, most reviewers cannot independently check results, and the authors themselves may be unable to recompute their own results in the future, given the impact of irreversible changes in the environment (compilers, libraries, components, etc.). The various forms of Artifact Evaluation improve confidence that computational results from these special platforms are correct.

The paper text explains why I believe my answers are right and shows all my work. Why do I need an Artifact Evaluation?

There are many good reasons for formalizing the artifact description and evaluation process. Standard practice varies across disciplines. Labeling the evaluation as such improves our ability to review the paper and improves reader confidence in the veracity of the results.

ARTIFACTS

What are “author-created” artifacts and why make the distinction?

Author-created artifacts are the hardware, software, or data created by the paper’s authors. Only these artifacts need to be made available to facilitate evaluation. Proprietary, closed-source artifacts (e.g. commercial software and CPUs) will necessarily be part of many research studies. These proprietary artifacts should be described to the best of the author’s ability but do not need to be provided.

What about proprietary author-created artifacts?

The ideal case for reproducibility is to have all author-created artifacts publicly available with a stable identifier. Papers involving proprietary, closed-source author-created artifacts should indicate the availability of the artifacts and describe them as much as possible. Note that results dependent on closed-source artifacts are not reproducible and are therefore ineligible for some artifact review badges.

Are the numbers used to draw our charts a data artifact?

Not necessarily. Data artifacts are the data (input or output) required to reproduce the results, not necessarily the results themselves. For example, if your paper presents a system that generates charts from datasets then providing an input dataset would facilitate reproducibility. However, if the paper merely uses charts to elucidate results, the input data to whatever tool you used to draw those charts aren’t required to reproduce the paper’s results. The tool which drew the chart isn’t part of the study, so the input data to that tool is not a data artifact of this work.

Help! My data is HUGE! How do I make it publicly available with a stable identifier?

Use Zenodo (https://help.zenodo.org). Contact them for information on how to upload extremely large datasets. You can easily upload datasets of 50GB or less, have multiple datasets, and there is no size limit on communities.

ARTIFACT REVIEW BADGES

The specific badges available to be awarded to participants will be forthcoming, check back soon for further information.

FAQ

Q: I only want to make my artifacts public if the paper is accepted. Is that okay?

A: Yes. Artifacts will only be examined after papers are accepted. You can wait until you hear the decision on your paper before making your artifacts public, but you must complete the form by the deadline set; information in the form will not be used until after paper acceptance.

Q: Does the artifact submission link need to be anonymous?

A: No, it should not be anonymous and thus should not be included in the submitted paper.  Any information in the paper must be anonymous to preserve the double-blind review.  After a paper has been accepted, links to the appropriate artifacts, code repositories, etc., can be added to the camera-ready paper.  The submitted Artifact Form is not anonymous and is reviewed independently and will not be examined until after paper decisions are made.

Q: What is the deadline for the submission of artifacts?

A: No one will look at the artifacts until your paper is accepted. We require the information available to start the evaluation process as soon as papers are accepted, hence the requirement to submit the form.

Q: Can I update the artifact evaluation form after paper acceptance?

A: You are free to make changes to the details of the artifacts up to the start of the evaluation and in consultation with your evaluator, but the initial submission should include as many details as possible to allow the evaluation process to be organized.