diff --git a/AD-AE-Appendices_Authors.md b/AD-AE-Appendices_Authors.md index d6ef1ce..961ed61 100644 --- a/AD-AE-Appendices_Authors.md +++ b/AD-AE-Appendices_Authors.md @@ -19,49 +19,72 @@ Artifact Evaluation (AE) Appendices are optional, but strongly encouraged. ## AD and AE Appendix Requirements -**Q1. Are AD and AE appendices required in order to submit to SC19?** +**Are AD and AE appendices required in order to submit to SC19?** -A1. An AD appendix is _required_ for all Technical Program submissions. +An AD appendix is _required_ for all Technical Program submissions. An AE appendix is _optional_ but strongly encouraged. -**Q2. Do I need to make my software open source in order to complete the AD appendix?** +**Do I need to make my software open source in order to complete the AD appendix?** -A2. No. You are not asked to make any changes to your computing environment in order to complete the appendix. +No. You are not asked to make any changes to your computing environment in order to complete the appendix. The AD appendix is meant to _describe_ the computing environment in which you produced your results. Any author-created software _does_ need to be open source, however, to be eligible for the ACM Artifacts Available badge (see below). -**Q3. How should I format my AD Appendix** +**How should I format my AD Appendix** -Q3. You don't need to worry about formatting the Appendices. You will be presented with an online form during the paper submission with questions that you will answer directly on the submission site. After answering the questions, the system will automatically generate a PDF of the Appendix for you. +You don't need to worry about formatting the Appendices. You will be presented with an online form during the paper submission with questions that you will answer directly on the submission site. After answering the questions, the system will automatically generate a PDF of the Appendix for you. -**Q4. What information do I need to provide in the AD/AE Appendix online form?** +**What information do I need to provide in the AD/AE Appendix online form?** -Q4. A printout of the questions included in the AD/AE Appendix online form is provided in the [`Author-Kit`](https://github.com/SC-Tech-Program/Author-Kit) repository. Be sure to familiarize yourself with these _before writing your paper_, and ideally before or while you are producing your results. +A printout of the questions included in the AD/AE Appendix online form is provided in the [`Author-Kit`](https://github.com/SC-Tech-Program/Author-Kit) repository. Be sure to familiarize yourself with these _before writing your paper_, and ideally before or while you are producing your results. ## Review Process -**Q5. Who will review my appendices?** +**Who will review my appendices?** -A5. The AD & AE Appendices will be reviewed _with your paper_ by the Technical Program committee, but the artifact URLs will be removed from the version they review, as a precaution in support of double-blind review. +The AD & AE Appendices will be reviewed _with your paper_ by the Technical Program committee, but the artifact URLs will be removed from the version they review, as a precaution in support of double-blind review. In addition, the **AD/AE Appendices Committee** will review the unredacted appendices, and will check that artifacts are indeed available in the URLs provided. They will also help authors improve their appendices, in a double-open arrangement. -**Q6. How will review of appendices interact with the double-blind review process?** +**How will review of appendices interact with the double-blind review process?** -A6. The AD appendix should describe the data, software and hardware artifacts involved in producing the results. +The AD appendix should describe the data, software and hardware artifacts involved in producing the results. Reviewers _could_ discover the author's identity if they embark on an online search, but they will be asked _not to_, in support of double-blind review. Author-provided artifact URLs will be redacted from the appendices provided to the reviewers. ## Impact of AD and AE Appendices -**Q7. What's the impact of an Artifact Description appendix on scientific reproducibility?** +**What's the impact of an Artifact Description appendix on scientific reproducibility?** -A7. Reproducibility depends on, as a first step, sharing the provenance of results with transparency, and the AD appendix is an instrument of documentation and transparency. A good AD appendix helps researchers document their results, and helps other researchers build from them. +Reproducibility depends on, as a first step, sharing the provenance of results with transparency, and the AD appendix is an instrument of documentation and transparency. A good AD appendix helps researchers document their results, and helps other researchers build from them. -**Q8. What's the impact of an Artifact Evaluation appendix on scientific reproducibility?** +**The paper text explains why I believe my answers are right and shows all my work. Why do I need to provide an Artifact Evaluation appendix?** -A8. An artifact-evaluation effort can increase the trustworthiness of computational results. +There are many good reasons for formalizing the artifact evaluation process. Standard practice varies across disciplines, and SC is an international, multi-disciplinary conference. Labeling the evaluation as such improves our ability to review the paper and improves reader confidence in the veracity of the results when approaching the work from a different background. + +## Artifacts + +**What are "author created" artifacts and why make the distinction?** + +Author created artifacts are the hardware, software, or data created by the paper's authors. Only these artifacts need be made available to facilitate reproducibility. Proprietary, closed source artifacts (e.g. commercial software and CPUs) will necessarily be part of many research studies. These proprietary artifacts should be described to the best of the author's ability but do not need to be provided. + +**What about proprietary author-created artifacts?** + +The ideal case for reproducibility is to have all author created artifacts publically available with a stable identifier. Papers involving proprietary, closed source author-created artifacts should indicate the availability of the artifacts and describe them as much as possible. Note that results dependent on closed source artifacts are not reproducible and are therefore ineligible for most of the ACM's artifact review badges. See https://www.acm.org/publications/policies/artifact-review-badging. + +**Are the numbers used to draw our charts a data artifact?** + +Not necessarily. Data artifacts are the data (input or output) required to reproduce the results, not necessarily the results themselves. For example, if your paper presents a system that generates charts from datasets then providing an input dataset would facilitate reproducibility. However, if the paper merely *uses charts to elucidate results* then the input data to whatever tool you used to draw those charts isn't required to reproduce the paper's results. The tool which drew the chart isn't part of the study, so the input data to that tool is not a data artifact of this work. + +**Help! My data is HUGE! How do I make it publically available with a stable identifier?** + +Use Zenodo (https://help.zenodo.org/). Contact them for information on how to upload extremely large datasets. You can easily upload datasets of 50GB or less, have multiple datasets, and there is no size limit on communities. + + +**What's the impact of an Artifact Evaluation appendix on scientific reproducibility?** + +An artifact-evaluation effort can increase the trustworthiness of computational results. It can be particularly effective in the case of results obtained using specialized computing platforms, not available to other researchers. Leadership computing platforms, novel testbeds, and experimental computing environments are of keen interest to the supercomputing community. Access to these systems is typically limited, however. Thus, most reviewers cannot independently check results, and the authors themselves may be unable to recompute their own results in the future, given the impact of irreversible changes in the environment (compilers, libraries, components, etc.). -The various forms of Artiface Evaluation improve confidence that computational results from these special platforms are correct. +The various forms of Artifact Evaluation improve confidence that computational results from these special platforms are correct. ## ACM Artifacts Available and Artifacts Evaluated Badges diff --git a/Panels_Authors.md b/Panels_Authors.md index 8a16242..bb205e2 100644 --- a/Panels_Authors.md +++ b/Panels_Authors.md @@ -11,7 +11,7 @@ Panel Format **Q: What is an SC panel?** -A: A panel provides a forum that promotes lively, highly interactive discussions on a wide variety of topics among the panelists and audience. The panel format includes a moderator and up to six panelists. A panel is not a set of mini-presentations that do not leave time for discussion with the audience. A panel is non-commercial. +A: A panel provides a forum that promotes lively, highly interactive discussions on a wide variety of topics among the panelists and audience. The panel format includes a moderator and panelists. We recommend no more than six panelists as this is the maximum number of chairs available for panelists. A panel is not a set of mini-presentations that do not leave time for discussion with the audience. A panel is non-commercial. **Q: What is the length of a panel session?** @@ -22,7 +22,7 @@ Submissions and Notifications **Q: When will I know if my proposed panel will be included in the conference?** -A: Accept or reject notifications will be sent to all submitters by late June. +A: Accept or reject notifications will be sent to all submitters by 22 June. **Q: Where can I find the Panels program online?** @@ -32,20 +32,25 @@ A: The program for Panels will be posted here within two weeks of the notificati **Q: Can I submit more than one panel proposal?** -A: You are welcome to submit multiple proposals. A committee of peers selects panels in a peer-review process. Because a limited number of slots are available, any individual may be a member (i.e., contact, moderator, or panelist) of at most two panels. +A: You are welcome to submit multiple proposals. A committee of peers selects panels in a peer-review process. Because a limited number of slots are available, any individual may be a member (i.e., contact, moderator, or panelist) of at most two accepted panels. **Q: Do panels need to have a title?** -A: You should indicate your panel’s title when submitting your proposal, as well as the name and affiliation of each discussant and the moderator. You do not have to include titles for individual presentations. A contact for the panel proposal should also be provided. You must include a brief abstract, in the form of a call for participation, describing the panel topic and questions to be addressed exactly as you want them to appear in the conference program. +A: Yes. You must indicate your panel’s title when submitting your proposal, as well as the name and affiliation of each panelist and the moderator. A contact for the panel proposal should also be provided. You must include a brief abstract, in the form of a call for participation, describing the panel topic and questions to be addressed exactly as you want them to appear in the conference program. **Q: What material do I need to submit?** A. Your submission should include: -* A brief abstract, in the form of a call for participation, describing the panel topic and questions to be addressed exactly as you want them to appear in the conference program; -* A proposal of no more than two pages, uploaded as a single PDF file, that describes the panel in more detail. This proposal should also include your panel title. The panel proposal may include information such as a description of the panel topic or position statements by the panelists. You must convince the Panels review committee that the panel will truly be an interactive session and not deteriorate into long-winded, disjointed, and boring mini-presentations or fluffy entertainment. Do not waste space giving a technical history; instead, explain why the panel will attract a good audience and why the audience will find the panel to be insightful and impactful. +* A title; +* Moderator information with a short biography (maximum 150 words); +* Contact person information if different from the moderator; +* Panelist information with a short biography (maximum 150 words); +* A brief abstract, in the form of a call for participation, describing the panel topic and questions to be addressed exactly as you want them to appear in the conference program (maximum 150 words); +* A proposal of no more than two pages, uploaded as a single PDF file, that describes the panel in more detail. This proposal should also include your panel title. The panel proposal may include information such as a description of the panel topic or position statements by the panelists. You must convince the Panels review committee that the panel will truly be an interactive session and not deteriorate into long-winded, disjointed, and boring mini-presentations or fluffy entertainment. Do not waste space giving a technical history; instead, explain why the panel will attract a good audience and why the audience will find the panel to be insightful and impactful; +* A diversity justification, describing how you plan to create diversity among panelists with respect to topical background, institution type, geography and demographic characteristics like seniority and gender (maximum 70 words). Review Criteria and Conflict of Interest (COI) ---------------- @@ -108,7 +113,7 @@ Infrastructure and Organization **Q: What infrastructure is provided for SC panels?** -A: Each panel’s infrastructure includes one projector, one screen, one table for the panelists with multiple chairs, one clip-on or one podium microphone for the moderator, and microphones for the panelists. One or multiple aisle microphone will be provided. The moderator should direct attendees to the aisle microphone for questions so that everyone can hear. +A: Each panel’s infrastructure includes one projector, two screens, two tables for the panelists with six chairs, one fixed microphone at the podium, a lavalier lapel microphone, and three or four microphones on the panelist tables. A stand microphone is provided in each of the three main aisles in the room. **Q: I am a panelist or a moderator. When should I arrive at and leave my designated room?** diff --git a/Papers_Authors.md b/Papers_Authors.md index 5da3710..68fe350 100644 --- a/Papers_Authors.md +++ b/Papers_Authors.md @@ -11,18 +11,22 @@ Paper formatting **Q: What are the length restrictions on paper submissions?** -A: Papers are limited to 10 pages, including everything except the References section. In particular, figures, tables, and any appendices included in the paper count against the 10 pages. +A: Submissions are limited to ten pages of 8.5" × 11" paper (i.e., U.S. letter). This ten-page limit includes everything except the bibliography. Authors of accepted papers can provide supplemental material with their final, camera-ready version of the paper (e.g., additional proofs, videos, or images). -**Q: Do the AD and AE appendices count against the 10 pages?** +**Q: Do the AD and AE appendices count against the ten pages?** A: No. These are created separately via an HTML form and are not considered part of the paper proper. +**Q: Can I include appendices other than the AD/AE appendices that do not count against the ten pages?** + +A: For the submission, no. The reviewing time is too limited for reviewers to have to evaluate more than ten pages per submission. However, authors of accepted papers will have the opportunity to upload separate, supplemental material to accompany the digital version of their paper. + **Q: How should papers be formatted?** A: For consistency across the entire proceedings, papers must be written in [ACM SIG proceedings style](https://www.acm.org/publications/proceedings-template) with line numbering enabled to help with the review process. In LaTeX, this implies a document class of ```LaTeX -\documentclass[sigconf,review]{acmart} +\documentclass[sigconf,review,anonymous]{acmart} ``` Contribution Track @@ -30,7 +34,7 @@ Contribution Track **Q: Is it mandatory for authors to select the primary track of their contribution?** -A: Yes. Authors must indicate their primary contribution track from the ten track choices on the submissions form. We understand that contributions may straddle more than one track. In such cases, we encourage authors to indicate a secondary contribution track. +A: Yes. Authors must indicate a primary track from the ten choices on the submissions form and are strongly encouraged to indicate a secondary track. **Q: Who will review my paper? The program committee corresponding to my primary contribution track? My secondary track? Both? Neither?** @@ -45,7 +49,29 @@ A: Yes. Papers that have not previously been peer-reviewed are eligible for sub **Q: If my paper was already published in a workshop, can I also submit it to SC?** -A: No, unless a paper has been substantially enhanced since prior publication (say, 30% new material), it is not eligible for submission to SC. +A: Yes. However the paper needs to have been substantially enhanced since prior publication (say, 30% new material) to be eligible for submission to SC. + +**Q: Can I submit a paper that is in submission to another venue?** + +A: No. If your paper has been submitted to a venue whose notification overlaps with the SC19 deadline (e.g. ICS 2019), you can only submit your paper to SC19 *after* withdrawing your paper from consideration at that other venue. + +**Q: My paper presents results from only a single GPU, workstation, etc. Does that make my paper ineligible for inclusion in SC?** + +A: Submissions will be considered on any topic related to high performance computing within Technical Papers’s ten tracks: + +* Algorithms +* Applications +* Architecture and Networks +* Clouds and Distributed Computing +* Data Analytics, Visualization, and Storage +* Machine Learning and HPC +* Performance Measurement, Modeling, and Tools +* Programming Systems +* State of the Practice +* System Software + +(See https://sc19.supercomputing.org/submit/paper-submissions/ for descriptions of each of the above.) Small-scale studies—including single-node studies—are welcome as long as the paper clearly conveys the work’s contribution to high-performance computing. + The Peer-Review Process ----------------------- @@ -54,22 +80,25 @@ The Peer-Review Process A: From an author’s perspective, the following are the key steps: -1. Authors submit a title, abstract, and other metadata. These are used to assign papers to appropriate reviewers and avoid conflicts of interest. +1. Authors submit a title, abstract, and other metadata. 2. Authors submit their full paper and complete a form describing their computational artifacts (or lack of computational artifacts) and, optionally, text discussing how they evaluated their computational results. + * Papers that violate any of the submission rules (e.g., page length) will be rejected without review. See "How many reviewers will review my paper?" below. 3. Authors receive an initial set of reviews of their paper. + * Very low-scoring papers will be rejected at this stage. 4. Authors have an opportunity to revise their paper and prepare an accompanying response to the reviewers. 5. Author revisions and accompanying response will be available to the reviewers at least a week before the PC meeting. -6. Authors are notified of their paper’s disposition: Accept, Reject, or Major Revisions Required. -7. In the case of Major Revisions Required, authors prepare a major revision for re-review. After this review, the paper will be either accepted or rejected. -8. Authors of accepted papers prepare the final version of their paper. + * Authors are notified of their paper’s disposition: Accept, Reject, or, in very few instances, Major Revisions Required. +6. In the case of Major Revisions Required, authors prepare a major revision to address a list of concerns provided with the review. + * If all of the concerns on the list are addressed, the paper will be accepted. If not, it will be rejected. +7. Authors of accepted papers prepare the final version of their paper. **Q: Is there an author response/rebuttal stage?** -A: Once authors receive their initial feedback from the reviewers, they are given the opportunity to improve their paper accordingly before the next round of reviews. For example, if a reviewer indicates that a paper requires some critical measurement or explanation to be worthy of publication, the authors can supply that missing information. An accompanying response document highlights the changes the authors made. This is unlike the rebuttal stage present in many other conferences in that the focus is less on promises to fix problems for the final document and more on actually fixing the problems. +A: Yes. Once authors receive their initial feedback from the reviewers, they are given the opportunity to improve their paper accordingly before the next round of reviews. For example, if a reviewer indicates that a paper requires some critical measurement or explanation to be worthy of publication, the authors can supply that missing information. An accompanying response document highlights the changes the authors made. This is unlike the rebuttal stage present in many other conferences in that the focus is less on promises to fix problems for the final document and more on actually fixing the problems. **Q: If I don’t have time to gather the additional data the reviewers requested by the revise-and-respond deadline but can do so by the final paper deadline, can my response indicate that.** -A: Yes. The reviewers may want to see a major revision in this case to ensure that no information critical to the paper’s argument is missing. +A: Yes. Depending on what other concerns reviewers have with the paper, it may still be rejected. Another possibility is that reviewers may want to see a major revision in this case to ensure that no information critical to the paper’s argument is missing. **Q: Now that I’ve read the reviews of my paper, I see much better how to organize it so it will be clear to the reader. Can I do this reorganization and upload the new version during the revise-and-respond period?** @@ -81,15 +110,29 @@ A: Yes, but you must clearly document your changes and rationale in your author **Q: Doesn’t this incentivize authors to submit an extremely rough first draft, and only make updates after reviewer feedback?** -A: Perhaps, but as the old adage goes, there’s no second chance to make a first impression. Your reviewers will have had much more time to look at your first draft, and their initial scores will be based on it. SC introduced the revise-and-respond process on the belief that an actual revision is far more persuasive than promises made in a rebuttal. That does not mean that revisions will change a reviewer’s mind if their first impression of a paper was completely negative. Because reviewers will have limited time to scrutinize your revisions, you should strive to make every phase of the submission process count. +A: No, because very low-scoring papers will be rejected after the first round of reviews, and authors will not be given an opportunity to restructure their prose, add data, or make other such improvements. As the old adage goes, there’s no second chance to make a first impression. **Q: What if a reviewer clearly didn’t read my paper carefully enough? What if the reviewer seems to lack basic knowledge of the area on which the paper focuses? How should my author response and revision address these issues?** -A: We’ve all received reviews that made us angry, particularly on first reading. The revision period is short and doesn’t allow for the cooling-off period that authors have before they write a response to a journal review. We provide you the opportunity to submit a revision and response to address these issues, but you should still be careful with the wording of your response. For example, we recommend you don’t say: “If reviewer X had just taken the time to read my paper carefully, they would have realized that our algorithm was rotation invariant.” Instead say, “Unfortunately, Section 4 must not have been as clear as we had hoped because Reviewer X did not understand that our algorithm was rotation invariant and they were therefore skeptical about the general applicability of our approach. The revised version of the second paragraph in Section 4 should clear up this confusion.” The revise-and-respond process offers more opportunity than what is present in other conferences to convince your reviewers of the merits of your work; use such opportunities wisely. +A: We’ve all received reviews that made us angry, particularly on first reading. The revision period is short and doesn’t allow for the cooling-off period that authors have before they write a response to a journal review. We provide you the opportunity to submit a revision and response to address these issues, but you should still be careful with the wording of your response. For example, we recommend you don’t say: “If reviewer *X* had just taken the time to read my paper carefully, they would have realized that our algorithm was rotation invariant.” Instead say, “Unfortunately, Section 4 must not have been as clear as we had hoped because Reviewer *X* did not understand that our algorithm was rotation invariant and they were therefore skeptical about the general applicability of our approach. The revised version of the second paragraph in Section 4 should clear up this confusion.” The revise-and-respond process offers more opportunity than what is present in other conferences to convince your reviewers of the merits of your work; use such opportunities wisely. + +**Q: How many reviewers will review my paper?** + +A: Each paper will normally receive a minimum of three reviews. However, the Technical Papers committee reserves the right to “desk reject” (i.e., reject without review) papers that violate any of the paper-submission rules. The following violations are leading grounds for desk rejects: + +* The paper is longer than 10 pages (excluding the bibliography). (See [Paper formatting](#paper-formatting).) + +* The paper is not formatted in ACM SIG proceedings style. (See [Paper formatting](#paper-formatting).) + +* The paper exposes the authors’ identities. (See [Double-blind reviewing](#double-blind-reviewing).) + +* The AD appendix claims that the paper has no associated author-created artifacts even though the paper obviously contains empirical results, discusses author-written software, presents author-created hardware, or employs author-generated data. The “no author-created artifacts” option in the AD form does not mean “I ran out of time to describe my artifacts.” (See [Artifact Description (AD) and Artifact Evaluation (AE) Appendices](#artifact-description-ad-and-artifact-evaluation-ae-appendices).) Double-blind reviewing ---------------------- +See the [official SC19 double-blind policy](https://sc19.supercomputing.org/submit/paper-submissions/double-blind-review-policy/) for more complete information. + **Q: What is double-blind reviewing?** A: Double-blind reviewing means that not only do authors not know the identities of their paper’s reviewers, but reviewers don’t know the identities of the paper’s authors until after acceptance decisions have been made. Reviewers will not learn the identity of the authors for rejected papers. @@ -98,13 +141,12 @@ A: Double-blind reviewing means that not only do authors not know the identities A: Studies indicate that double-blind reviewing goes a long way in reducing unconscious bias (e.g., based on institution, seniority, nationality, or gender) on the part of the reviewers. For more information, here are some entry points into the relevant literature: -* “More on Improving Reviewing Quality with Double-Blind Reviewing, External Review Committees, Author Response, and in Person Program Committee Meetings” by McKinley (2015), http://www.cs.utexas.edu/users/mckinley/notes/blind.html -* “Reviewer bias in single- versus double-blind peer review” by Tomkins (2017), https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5715744/ - +* “[More on Improving Reviewing Quality with Double-Blind Reviewing, External Review Committees, Author Response, and in Person Program Committee Meetings](http://www.cs.utexas.edu/users/mckinley/notes/blind.html)” by McKinley (2015) +* “[Reviewer bias in single- versus double-blind peer review](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5715744/)” by Tomkins (2017) **Q: How should I prepare my paper for double-blind review?** -A: SC requires only that there be some level of doubt as to the authors’ identities, not that only intelligence agencies could possibly determine who the authors are. Consequently, there are only two key pieces to SC’s double-blind policy: +A: SC requires only that there be some level of doubt as to individual authors’ identities, not that only government intelligence agencies could possibly determine who the authors are. Consequently, there are only two key pieces to SC’s double-blind policy: 1. Don’t state the authors’ names, institutions, or other identifying information anywhere in the paper. 2. Write in the third person about your prior work, software, etc. @@ -115,6 +157,8 @@ In particular, redact nothing: * Don’t obscure the hardware platforms you used/built, even if relatively few people have access to that hardware. * Don’t obscure the software you used/developed, even if relatively few people have access to that software. +As a corollary, don’t substitute fake names for names of software, hardware, institutions, people, etc. + For example, consider the following alternatives: > **Bad**: We enhanced our INTERCAL compiler [1] to accept keywords in Pig Latin. @@ -146,19 +190,19 @@ A: They could, but reviewers are asked to avoid trying to learn the authors’ i Artifact Description (AD) and Artifact Evaluation (AE) Appendices ----------------------------------------------------------------- +See the [AD/AE Appendices FAQ](AD-AE-Appendices_Authors.md) for more complete information. + **Q: What is the AD appendix?** A: The AD appendix, generated by the SC submission site from content gathered via a form, lets authors share with readers a description of the data, software, and hardware artifacts relied on to produce the paper’s results and provide links to persistently archived data and software products. -See the [AD/AE Appendices FAQ](AD-AE-Appendices_Authors.md) for complete information. - **Q: Does writing an AD appendix imply I can skip/shorten my paper’s experimental-setup or or other such sections?** A: An SC paper must stand on its own and will be reviewed as such. The AD appendix provides additional information to reviewers and readers, helping them better interpret the results and helping other researchers build on those results. **Q: What is the AE appendix?** -A: The AE appendix, generated by the SC submission site from a free-text form field, lets authors explain the trustworthiness of their data by detailing: their approach to verification and validation, statistics gathered, uncertainty-quantification techniques applied, conditions controlled/not controlled for, and other details that convey a sense of the confidence on the reported results. This is especially useful for results obtained in computational environments that are not widely accessible and/or an environments that are not reproducible. +A: The AE appendix, generated by the SC submission site from a free-text form field, lets authors explain the trustworthiness of their data by detailing their approach to verification and validation, statistics gathered, uncertainty-quantification techniques applied, conditions controlled/not controlled for, and other details that convey a sense of the confidence on the reported results. This is especially useful for results obtained in computational environments that are not widely accessible and/or environments that are not easily reproducible. **Q: Are the AD and AE appendices mandatory or optional?** @@ -166,27 +210,27 @@ A: Starting with SC19, the AD appendix is mandatory for all papers in the Techni **Q: What if our paper does not have any associated artifacts?** -A: Although we expect this to be a rare occurrence for SC papers, the first question in the required Artifact Description form enables authors to indicate that they do not have any software, hardware, or data artifacts associated with the submitted paper. +A: Although we expect this to be an exceptionally rare occurrence for SC papers, the first question in the required AD form enables authors to indicate that their paper was written without having developed any software, run on any hardware, or gathered any data. **Q: Will the paper reviewers also be reviewing the AD and AE appendices?** -A: Yes. Their focus will be on the paper itself, but they can refer to the appendices for clarification and convincing of the reliability of the reported results. For compatibility with the double-blind review process, URLs linking to archived artifacts will be removed before review. +A: Yes. Their focus will be on the paper itself, but they can refer to the appendices for clarification and convincing of the reliability of the reported results. **Q: Doesn’t an artifact description conflict with double-blind reviewing?** -A: Since your paper should stand on its own without the AD appendix, the paper would already have a third-party citation to software and hardware the research builds on. Keep in mind that double-blind reviewing is for reducing the chances that a reviewer will be able to guess your identity. You do not have to guarantee it. Additionally, URLs to archived software and data products willif you provide a link to your software, that link will *not* be shared with PC members onlyuntil after the paper has been accepted. +A: For compatibility with the double-blind review process, URLs provided in the AD appendix form question asking for URLs will *not* be shared with PC members during the review process. Because your paper should stand on its own without the AD appendix, the paper will presumably already have a third-party citation to the software and hardware upon which the research builds. Keep in mind that double-blind reviewing is for reducing the chances that a reviewer will be able to guess your identity. You do not have to guarantee it. Plagiarism ---------- -**Q: I understand that the SC Conference applies a plagiarism test program to submissions. What constitutes plagiarism? Is it possible for an author to plagiarize their own work?** +**Q: I understand that the SC Conference applies a plagiarism test program to submissions. What constitutes plagiarism? Is it possible for authors to plagiarize their own work?** A: Please see [ACM’s guidelines on identifying plagiarism](https://www.acm.org/publications/policies/plagiarism). Authors should submit new, original work that represents a significant advance from even their own prior publications. Conflicts of Interest --------------------- -**Q: What are the SC guidelines for Conflicts of Interest (COI)?** +**Q: What are the SC guidelines for conflicts of interest (COI)?** A: A potential conflict of interest occurs when a person is involved in making a decision that 1) could result in that person, a close associate of that person, or that person’s company or institution receiving significant financial gain, such as a contract or grant, or 2) could result in that person, or a close associate of that person, receiving significant professional recognition, such as an award or the selection of a paper, work, exhibit, or other type of submitted presentation. @@ -206,6 +250,11 @@ Other situations can create COIs, and you should contact the Technical Papers Ch Specific technical tracks ------------------------- +**Q: What kind of papers is the Performance Measurement, Modeling, and Tools Track looking for?** + +“Performance” can be broadly construed to include any number of metrics, such as execution time, bandwidth, energy, power; it can also include measures of correctness and resilience. Techniques that exploit machine learning should have a primary focus on applying their techniques to analyze or enhance system performance. If their focus is more on the machine learning algorithm itself, then consider submitting to the new ML and HPC track. + +Submissions in to the Performance track are encouraged to show the applicability and reproducibility of their results by means such as sensitivity analysis, performance modeling, or code snippets. **Q: What constitutes a State of the Practice (SOP) paper?** diff --git a/Posters_Authors.md b/Posters_Authors.md index cb595c5..5b139b4 100644 --- a/Posters_Authors.md +++ b/Posters_Authors.md @@ -7,34 +7,21 @@ permalink: /posters-author/ Find answers to common questions that may arise during the effort to prepare and submit a poster and scientific visualization and data analytics showcases to the SC19 Conference. If your question is not addressed here, please contact us at posters@info.supercomputing.org. FAQ's are organized into the following sections: - - [Formatting Specifications](#Formatting) - - [Submissions and Review Process](#Submissions) - - [Mounting/Displaying at the Conference](#Mounting) - - [Reproducibility](#Reproducibility) + - [General Questions](#GeneralQuestions) + - [ACM Student Research Posters Questions](#ACMSRCposters) + - [Research Posters Questions](#Research) + - [Doctoral Showcase Questions](#Doctoral) + - [Scientific Visualization & Data Analytics Showcase Questions](#ScientificViz) -## Formatting Specifications +## General Questions +### Formatting Specifications +**Q. What are the formatting requirements for research posters, ACM SRC posters, and Doctoral Showcase thesis canvases?** -**Q. What are the poster formatting requirements for research and ACM SRC posters?** +A. Each poster/thesis canvas is assigned one display board that is four feet high and eight feet wide. The only format requirement is that your poster properly mounts on its display board. Your poster does not need to cover the entire display board; the poster size should be appropriate for your content. -A. Each poster is assigned one display board that is four feet high and eight feet wide. The only format requirement is that your poster properly mounts on its display board. +**Q. Is there a poster template for research posters, ACM SRC posters, and Doctoral Showcase thesis canvases?** -**Q. Is there a poster template for research and ACM SRC posters?** - -A. You should produce a poster that effectively and elegantly portrays its technical content. Poster styles, layouts, fonts, and content are left to the discretion of the author(s). - -**Q. What are the poster formatting requirements fors scientific visualization & data analytics showcase?** - -A. Submissions need to include a movie (up to 1GB in size) and a short paper (up to 6 pages including references). The short paper should describe the scientific story conveyed by the movie, how the visualization helps scientific discovery, and the “state-of-the-practice” information behind making the movie. - -The video should be in MP4, MOV, or AVI format and use a codec that will play in typical desktop computer media players. There is no hard limit on video length, but typical videos are less than 10 minutes long, and the submitted file should be no more than 1GB. - -The manuscript must be no more than 6 pages including references. The manuscripts should be formatted according to the guidelines for the Parallel Computing journal. Please use one of the following two templates when formatting your manuscript: - -LaTeX Template (159KB .zip) – preferred -Microsoft Word Template (51KB .zip) - - -The manuscript should be uploaded as a pdf for the initial submission. +A. You should produce a poster/thesis canvas that effectively and elegantly portrays its technical content. Poster styles, layouts, fonts, and content are left to the discretion of the author(s). **Q. Can I embed content in my poster?** @@ -54,13 +41,13 @@ A. There are many QR code-generating sites and apps, both paid and free, that yo **Q. Where do I place the QR codes on my poster?** -A. QR codes should be placed anywhere you want to link to external content on your poster; for example, to point to a video narration of the poster by the author, links to results, movies, graphics, datasets, codes, etc. +A. QR codes could be placed anywhere you want to link to external content on your poster; for example, to point to a video narration of the poster by the author, links to results, movies, graphics, datasets, codes, etc. **Q. I noticed QR codes next to _every_ poster at the conference. Do I have to generate that QR code for my poster?** A. No, you do not have to generate those QR codes. We generate one QR code for every poster that points to your submitted content. This code is attached to the poster pedestal by the time you arrive at SC. Conference attendees can scan these QR codes and get all of your material easily. -## Submissions and Review Process +### Submissions and Review Process **Q. Should I submit the final version of my poster for review? Can I make changes after the review?** @@ -68,56 +55,22 @@ A. For the submission, the poster does not need to be finalized; you may update **Q. Are posters peer-reviewed?** -A. Yes, all regular and ACM Student Research Competition posters are peer-reviewed by a committee of experts. +A. Yes, all research and ACM Student Research Competition posters and Doctoral Showcase thesis canvases are peer-reviewed by a committee of experts. **Q. Is my poster blind-reviewed?** A. Poster reviews are single-blind: reviewers will see author names, but authors will not see reviewer names. -**Q. Should I be the only author of my ACM Student Research Competition poster?** - -A. An ACM Student Research Competition submission reflects student research: this work may part of a larger project, but the poster should only reflect the contributions of a single graduate student or up to two undergraduate students. For graduate student submissions, the final poster must list one student as the only author; for undergraduate student submissions, the final poster must list up to two undergraduate students as the first (and second) author(s). - -Students are allowed to list one or more official graduate or undergraduate advisors on their poster. Each advisor must be clearly marked by putting “(advisor)” after the advisor’s name. For example, if Alex Abel is advised by Bruno Banach and Clara Cantor at Durham University in England, then the poster and extended abstract should be authored by: Alex Abel, Bruno Banach (advisor), and Clara Cantor (advisor), Durham University, England. - -**Q. How is the Best Poster for Research Posters selected?** - -A. The Research Posters Committee nominates Best Poster candidates during the review process. An ad-hoc committee judges posters and poster presentations during the Posters Reception. After the reception, this committee convenes and selects a single best poster. - -A separate award committee handles Best Student Poster in the context of the ACM Student Research Competition. **Q. Can I change from one type of poster to another?** -A. No. Authors are required to select the type of poster at the time of submission. Authors can select either Regular Poster or ACM Student Research Competition Poster. - -**Q. Can I submit a poster on HPC education?** +A. No. Authors are required to select the type of poster at the time of submission. Authors can select either Research Poster, ACM Student Research Competition Poster, Doctoral Showcase canvas, or Scientific Visualization movie. -A. Yes. Posters focusing on HPC education are within scope. This is one of the topics listed in the call for posters. There is no separate education track, so education posters will be evaluated as research posters and should be submitted using the same submission form as technical research posters. -Archiving/Publishing **Q. Is my poster archived? What is the process for archiving my poster?** A. Yes, all posters are archived and can be accessed after the conference on the SC19 website. Contact us for further details. -**Q. What is the submission process for Scientific Visualization and what is the review process?** - -A. Submissions need to include a movie (up to 1GB in size) and a short paper (up to 6 pages including references). The short paper should describe the scientific story conveyed by the movie, how the visualization helps scientific discovery, and the “state-of-the-practice” information behind making the movie. - -Each submission will be peer reviewed by the Visualization & Data Analytics Showcase Committee. Criteria for review include: - -How effective is the visual communication of the data? -How relevant to the HPC community is the visualization? -What is the impact of the science story and how well is it told? -What visualization techniques were necessary to create the movie? -Finally, submissions should consider SC19’s overall theme “HPC is now!”. - -Entries are submitted through the SC [Submissions website](https://submissions.supercomputing.org): - -Sign in -Click the “Make a New Submission” tab -Select “Scientific Visualization & Data Analytics Showcase” - - ## Mounting/Displaying at the Conference **Q. What are the options for mounting my poster to the display board?** @@ -136,18 +89,53 @@ A. Each poster is given a designated space with a display board. Before the conf A. The conference does not provide any support for printing posters. Typically, authors print their posters ahead of the conference. The convention center’s business office may offer printing services. You can look for details at the convention center’s website. +**Q. When do I need to be at my poster location to present it?** +A. Poster presenters must be available at their posters to answer questions and discuss their work for the duration of the Posters Reception on Tuesday evening. In addition, each poster track has a separate presentation session for selected posters (Research Best Poster Candidates, ACM SRC Best Poster Candidates, Doctoral Showcase, Best Poster Candidates in Scientific Visualization). -## Reproducibility -**Q. Is an artifact description appendix required in order to make a submission to SC Posters?** +## ACM Student Research Competition Posters -A. The appendix is not required and is an option for Research Posters only. +**Q. Should I be the only author of my ACM Student Research Competition poster?** + +A. An ACM Student Research Competition submission reflects student research: this work may part of a larger project, but the poster should only reflect the contributions of a single graduate student or up to two undergraduate students. For graduate student submissions, the final poster must list one student as the only author; for undergraduate student submissions, the final poster must list up to two undergraduate students as the first (and second) author(s). + +Students are allowed to list one or more official graduate or undergraduate advisors on their poster. Each advisor must be clearly marked by putting “(advisor)” after the advisor’s name. For example, if Alex Abel is advised by Bruno Banach and Clara Cantor at Durham University in England, then the poster and extended abstract should be authored by: Alex Abel, Bruno Banach (advisor), and Clara Cantor (advisor), Durham University, England. **Q. Is the artifact description appendix required in order to be considered a finalist in the ACM Student Research Competition?** A. No. The appendix is not required in order for a poster to be considered a finalist in the ACM Student Research Competition. +## Research Posters + +### General Questions for Research Posters + +**Q. How is the Best Poster for Research Posters selected?** + +A. The Research Posters Committee nominates Best Poster candidates during the review process. At the conference, an ad-hoc committee judges poster content, including the optional artifact description appendix, and poster presentations during a Best Posters presentation session and the Posters Reception. After the reception, this committee convenes and selects a single best poster. + +A separate award committee handles Best Student Poster in the context of the ACM Student Research Competition. + +**Q. Are Research Posters that submit an artifact description appendix eligible for an Artifacts Available badge as described on the ACM website?** + +A. No. Research posters are not eligible for any badges associated with reproduciblity artifacts. + +**Q. Can I submit a poster on HPC education?** + +A. Yes. Posters focusing on HPC education are within scope. This is one of the topics listed in the call for posters. There is no separate education track, so education posters will be evaluated as research posters and should be submitted using the same submission form as technical research posters. +Archiving/Publishing + +**Q. Which ACM template should I use? There are two at the link you provided.** + +A. Please use the ACM Conference Proceedings "Master" Template for Research Posters abstracts. + +### Reproducibility for Research Posters + +**Q. Is an artifact description appendix required in order to make a submission to SC Posters?** + +A. The appendix is not required and is an option for Research Posters only. + + **Q. Do I need to make my software open-source in order to complete the artifact description appendix?** A. No. It is not required that you make any changes to your computing environment in order to complete the appendix. The artifact description appendix is meant to provide information about the computing environment you used to produce your results, reducing barriers for future replication of your results. @@ -160,10 +148,47 @@ A. The artifact description appendix will be submitted at the same time as your A. The artifact description appendix is simply a description of the computing environment used to produce the results in your poster. By itself, this appendix does not directly improve scientific reproducibility. However, a well-written appendix can be used by scientists (including the authors at a later date) to more easily replicate and build upon the results in the poster. Therefore, the artifact description appendix can reduce barriers and costs of replicating published results. It is an important first step toward full scientific reproducibility. -**Q. Are Research Posters that submit an artifact description appendix eligible for an Artifacts Available badge as described on the ACM website?** - -A. No. Research posters are not eligible for any badges associated with reproduciblity artifacts. **Q. What kind of recognition do I get for submitting these appendices?** A. There is no additional recognition for research posters that submit an artifact description appendix. + + + + +## Doctoral Showcase + +**Q. Why is the poster for the Doctoral Showcase called "Thesis Canvas" and not simply "Poster"?** + +A. Thesis Canvas is meant to present the research during the PhD studies giving a broad and holistic perspective on the PhD thesis research questions, methodolody, results and their critical assesment. Instead, a poster presents a focused, specific research problem. + +## Scientific Visualization & Data Analytics Showcase + +**Q. What are the poster formatting requirements fors scientific visualization & data analytics showcase?** + +A. Submissions need to include a movie (up to 1GB in size) and a short paper (up to 6 pages including references). The short paper should describe the scientific story conveyed by the movie, how the visualization helps scientific discovery, and the “state-of-the-practice” information behind making the movie. + +The video should be in MP4, MOV, or AVI format and use a codec that will play in typical desktop computer media players. There is no hard limit on video length, but typical videos are less than 10 minutes long, and the submitted file should be no more than 1GB. + +The manuscript must be no more than 6 pages including references. The manuscripts should be formatted according to the guidelines for the Parallel Computing journal. Please use one of the following two templates when formatting your manuscript: + +[LaTeX Template](sc19-vis-showcase-latex-template.zip) – preferred +[Microsoft Word Template](sc19-vis-showcase-template.docx.zip) + +The manuscript should be uploaded as a pdf for the initial submission. + +**Q. What is the submission process for Scientific Visualization and what is the review process?** + +A. Submissions need to include a movie (up to 1GB in size) and a short paper (up to 6 pages including references). The short paper should describe the scientific story conveyed by the movie, how the visualization helps scientific discovery, and the “state-of-the-practice” information behind making the movie. + +Each submission will be peer reviewed by the Visualization & Data Analytics Showcase Committee. Criteria for review include: + +How effective is the visual communication of the data? +How relevant to the HPC community is the visualization? +What is the impact of the science story and how well is it told? +What visualization techniques were necessary to create the movie? +Finally, submissions should consider SC19’s overall theme “HPC is now!”. + +Entries are submitted through the SC [Submissions website](https://submissions.supercomputing.org). + + diff --git a/README.md b/README.md index f398620..1d2393b 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,5 @@ # FAQs -Frequently Asked Questions the reproducibility initiative of the SC Conference. - -## For authors -* [AD/AE Appendices FAQs](AD-AE-Appendices_Authors.md) +Frequently Asked Questions the Techical Program content of the SC Conference. ## Adding Content To add content to this website, create a pull request in this repository. diff --git a/Tutorials_Authors.md b/Tutorials_Authors.md new file mode 100644 index 0000000..bdfecbd --- /dev/null +++ b/Tutorials_Authors.md @@ -0,0 +1,55 @@ +--- +layout: page +title: Tutorials FAQ +permalink: /tutorials/ +--- + +Find answers to common questions that may arise during the effort to prepare and submit a tutorial to the SC Conference. If your question is not addressed here, please email to tutorials@info.supercomputing.org + +**Q: How should I format my tutorial proposal submission?** + +A: Each tutorial submission must present its course information through the SC submissions website, where a form will collect basic information (the tutorial’s title, length, presenter information, and keywords). A sample form is available at the submissions website. Further details about the proposal, such as goals, targeted audience, and outline, should be written in a PDF and uploaded as an attachment to the submission form. The format for the PDF is documented in the How to Submit section of the submissions website. Submitters are particularly encouraged to include a draft of their full tutorial presentation in the Sample of Visual Material section of the submission form. + +**Q: Can I make changes to my tutorial materials after I have submitted them?** + +A: Tutorial presenters are generally expected to use the material submitted, with no changes. Minor extensions may be granted to enable a presenter to include more timely material, or to comply with conditions beyond the presenter’s control (e.g., new decisions adopted by standards bodies). In such cases, the submitted material should be as close to a final version as possible, and clearly indicate where changes are anticipated. Also, the deadline for the submission of the material is only one week before the tutorial, so no major updates are expected. +Changes requested because a presenter had insufficient time to prepare and submit their materials cannot be accommodated. + +**Q: What are the expectations of tutorial presentations at the conference?** + +A: SC Tutorials are some of the most popular sessions at the conference, attracting several thousand participants. Tutorial abstracts must clearly present what lessons can be learned, and tutorial presenters are expected to deliver professional presentations and to treat tutorial attendees with respect. Each tutorial will be evaluated in detail by attendees after the session, and these evaluations will play a crucial role in the evaluation of future-year tutorial submissions. + + +**Q: Q: Is the peer-review process double-blind?** + +A: No. Reviewers have access to the names of tutorial submitters. While Tutorials Committee members are named on the SC19 Planning Committee page, the names of the individuals reviewing each proposal are not made available to the authors. + +**Q: What are the guidelines for conflicts of interest (COI)?** + +A: A potential conflict of interest occurs when a person is involved in making a decision that: + +1. Could result in that person, a close associate of that person, or that person’s company or institution receiving significant financial gain, such as a contract or grant. + +2. Could result in that person, or a close associate of that person, receiving significant professional recognition, such as an award or the selection of a paper, work, exhibit, or other type of submitted presentation. + +Authors and Tutorials Committee members will be given the opportunity to list any potential COIs during the submissions and review processes, respectively. The Tutorials Committee Chair will make every effort to avoid assignments that have a potential COI. + +According to the SC Conference you have a conflict of interest with: + +* Your PhD advisors, post-doctoral advisors, PhD students, and post-doctoral advisees forever; +* Family relations by blood or marriage, or equivalent (e.g., a partner); +* People with whom you collaborated in the past five years. Collaborators include: co-authors on an accepted/rejected/pending research paper; co-PIs on an accepted/pending grant; those who fund your research; researchers whom you fund; or researchers with whom you are actively collaborating; +* Close personal friends or others with whom you believe a conflict of interest exists; +* People who were employed by, or a student at, your primary institution(s) in the past five years, or people who are active candidates for employment at your primary institution(s). + +Note that “service” collaborations, such as writing a DOE, NSF, or DARPA report, or serving on a program committee, or serving on the editorial board of a journal do not inherently create a COI. + +**Q: What rooms and audio/video infrastructure are provided to each tutorial?** + +A: Tutorials are assigned either a classroom or theater room equipped with standard AV facilities (projector, microphone and podium, wireless lapel microphone or wireless handheld microphone, and projection screen). + +**Q: Will tutorials’ material be provided in a USB?** + +A: No. Tutorials’ materials will be available for download in a password protected repository. + + diff --git a/sc19-vis-showcase-latex-template.zip b/sc19-vis-showcase-latex-template.zip new file mode 100644 index 0000000..8b9b199 Binary files /dev/null and b/sc19-vis-showcase-latex-template.zip differ diff --git a/sc19-vis-showcase-template.docx.zip b/sc19-vis-showcase-template.docx.zip new file mode 100644 index 0000000..d4e3d00 Binary files /dev/null and b/sc19-vis-showcase-template.docx.zip differ