Opinion Column: Tracks, Reviews and Preliminary Works

 

Welcome to the first edition of the  SIGMM Community Discussion Column!

As promised in our introductory edition, this column will report highlights and lowlights of online discussion threads among the members of the Multimedia community (see our Facebook MM Community Discussion group).

After an initial poll, this quarter the community chose to discuss about the reviewing process and structure of the SIGMM-sponsored conferences. We organized the discussion around 3 main sub-topics: importance of tracks, structure of reviewing process, and value of preliminary works.  We collected more than 50 contributions from the members of the Facebook MM Community Discussion group. Therefore, the following synthesis represents only these contributions. We encourage everyone to participate in the upcoming discussions, so that this column becomes more and more representative of the entire community.

In a nutshell, the community agreed that: we need more transparent communication and homogeneous rules across thematic areas; we need more useful rebuttals; there is no need for conflict of interest tracks; large conferences must protect preliminary and emergent research works. Solutions were suggested to improve these points.

Communication, Coordination and Transparency. All participants agreed that more vertical (from chairs to authors) and horizontal (in between area chairs or technical program chairs) communication could improve the quality of both papers and reviews in SIGMM-sponsored conferences. For example, lack of transparency and communication regarding procedures might deal to uneven rules and deadlines across tracks.

Tracks. How should conference thematic areas be coordinated? The community’s view can be summarized into 3 main perspectives:

  1. Rule Homogeneity.  The majority of participants agreed that big conferences should have thematic areas, and that tracks should be jointly coordinated by a technical program committee. Tracks are extremely important, but in order for the conference to give an individual, unified message, as opposed to “multi-conferences”, the same review and selection process should apply to all tracks. Moreover, hosting a face to face global TPC meetings is key for a solid, homogeneous conference program.
  2. Non-uniform Selection Process to Help Emerging Areas. A substantial number of participants pointed out that one role of the track system is to help emerging subcommunities: thematic areas ensure a balanced programme with representation from less explored topics (for example, music retrieval or arts and multimedia). Under this perspective, while the reviewing process should be the same for all tracks, the selection phase could be non-uniform. “Mathematically applying a percentage rate per area” does not help selecting the actually high-quality papers across tracks: with a uniformly applied low acceptance rate rule, minor tracks might have one or two papers accepted only, despite the high quality of the submissions.
  3. Abolish Tracks. A minority of participants agreed that, similar to big conferences such as CVPR, tracks should be completely abolished. A rigid track-based structure makes it somehow difficult for authors to choose the right track where to submit; moreover, reviewers and area chairs are often experts in more than one area. These issues could be addressed by a flexible structure where papers are assigned to area chairs and reviewers based on the topic.

Reviewing process  How do we want the reviewing process to be? Here is the view of the community on four main points: rebuttal, reviewing instructions, conflict of interest, and reviewers assignment.

  1. Rebuttal: important, but we need to increase impact. The majority of participants agreed that rebuttal is helpful to increase review quality and to grant authors more room for discussion. However, it was pointed out that sometimes the rebuttal process is slightly overlooked by both reviewers and area chairs, thus decreasing the potential impact of the rebuttal phase. It was suggested that, in order to raise awareness on rebuttal’s value, SIGMM could publish statistics on the number of reviewers who changed their opinion after rebuttal. Moreover, proposed improvements on the rebuttal process included: (1) more time allocated for reviewers to have a discussion regarding the quality of the papers; (2) a post-rebuttal feedback where reviewers respond to authors’ rebuttal (to promote reviewers-authors discussion and increase awareness on both sides) and (3) a closer supervision of the area chairs.
  2. Reviewing Guidelines: complex, but they might help preliminary works. Do reviewing guidelines help reviewers writing better reviews? For most participants, giving instructions to reviewers appear to be somehow impractical, as reviewers do not necessarily read or follow the guidelines. A more feasible solution is to insert weak instructions through specific questions in the reviewing form (e.g. “could you rate the novelty of the paper?”). However, it was also pointed out that written rules could help area chairs justify a rejection of a bad review.  Also, although reviewing instructions might change from track to track, general written rules regarding “what is a good paper” could help the reviewers understand what to accept. For example, clarification is needed on the depth of acceptable research works, and on how preliminary works should be evaluated, given the absence of a short paper track.
  3. Brave New Idea Track: ensuring scientific advancement. Few participants expressed their opinion regarding this track hosting novel, controversial research ideas. They remarked the importance of such a track to ensure scientific advancement, and it was suggested that, in the future, this track could host exploratory works (former short papers), as preliminary research works  are crucial to make a conference exciting.
  4. Conflict of Interest (COI) Track: perhaps we should abolish it. Participants almost unanimously agreed that a COI track is needed only when the conference management system is not able to handle conflicts on its own. It was suggested that, if that is not the case, a COI track might actually have a antithetical effect (is the COI track acceptance rate for ACM MM higher this year?).
  5. Choosing Reviewers: A Semi-Automated Process. The aim of the reviewers assignment procedure is to give the right papers to the right reviewers. How to make this procedure successful? Some participants supported the “fully manual assignment” option, where area chairs directly nominate reviewers for their own track. Others proposed to have a “fully automatic assignment”, based on an automated matching system such as the Toronto Paper Matching System (TPMS). A discussion followed, and eventually most participants agreed on a semi-automated process, having first the TPMS surfacing a relevant pool of reviewers (independent of tracks) and then area chairs manually intervening. Manual inspection of area chairs is crucial for inter-disciplinary papers needing reviews from experts from different areas.

Finally, during the discussion, few observations and questions regarding the future of the community arouse. For example: how to steer the direction of the conference, given the increase in number of AI-related papers? How to support diversity of topics, and encourage papers in novel fields (e.g. arts and music) beyond the legacy (traditional multimedia topics)? Given the wide interest on such issues, we will include these discussion topics in our next pre-discussion poll. To participate in the next discussion, please visit and subscribe to the Facebook MM Community Discussion group, and raise your voice!

Xavier Alameda-Pineda and Miriam Redi.

Interview Column – Introduction

The interviews in the SIGMM records aim to provide the community with the insights, visions, and views from outstanding researchers in multimedia. With the interviews we particularly try to find out what makes these researchers outstanding and also to a certain extend what is going on in their mind, what are their visions and what are their thoughts about current topics. Examples from the last issues include interviews with Judith Redi, Klara Nahrstedt, and Wallapak Tavanapong.

The interviewers are conducted via Skype or — even better — in person by meeting them at conferences or other community events. We aim to publish three to four interviews a year. If you have suggestions for who to interview, please feel free to contact one of the column editors, which are:

Michael Alexander Riegler is a scientific researcher at Simula Research Laboratory. He received his Master’s degree from Klagenfurt University with distinction and finished his PhD at the University of Oslo in two and a half years. His PhD thesis topic was efficient processing of medical multimedia workloads.
His research interests are medical multimedia data analysis and understanding, image processing, image retrieval, parallel processing, gamification and serious games, crowdsourcing, social computing and user intentions. Furthermore, he is involved in several initiatives like the MediaEval Benchmarking initiative for Multimedia Evaluation, which runs this year the Medico task (automatic analysis of colonoscopy videos, http://www.multimediaeval.org/mediaeval2017/medico/.

DSC_0104

Herman Engelbrecht is one of the directors at the MIH Electronic Media Laboratory at Stellenbosch University. He is a lecturer in Signal Processing at the Department of Electrical and Electronic Engineering. His responsibilities in the Electronic Media Laboratory are the following: Managing the immediate objectives and research activities of the Laboratory; regularly meeting with postgraduate researchers and their supervisors to assist in steering their research efforts towards the overall research goals of the Laboratory; ensuring that the Laboratory infrastructure is developed and maintained; managing interaction with external contractors and service providers; managing the capital expenditure of the Laboratory; and managing the University’s relationship with the post­graduate researchers – See more at: http://ml.sun.ac.za/people/dr-ha-engelbrecht/#sthash.3SexKFo5.dpuf

herman

Mathias Lux is associate professor at the Institute for Information Technology (ITEC) at Klagenfurt University. He is working on user intentions in multimedia retrieval and production and emergent semantics in social multimedia computing. In his scientific career he has (co-) authored more than 80 scientific publications, has served in multiple program committees and as reviewer of international conferences, journals and magazines, and has organized multiple scientific events. Mathias Lux is also well known for the development of the award winning and popular open source tools Caliph & Emir and LIRe (http://www.semanticmetadata.net) for multimedia information retrieval. Dr. Mathias Lux received his M.S. in Mathematics 2004, his Ph.D. in Telematics 2006 from Graz University of Technology, both with distinction, and his Habilitation (venia docendi) from Klagenfurt University in 2013.

Mathias_Lux_2016

Introduction to the Opinion Column

Welcome to the SIGMM Community Discussion Column! In this very first edition we would like to introduce the column to the community, its objectives and main operative characteristics.

Given the exponential amount of multimedia data shared online and offline everyday, research in Multimedia is of unprecedented importance. We might be now facing a new era of our research field, and we would like the whole community to be involved in the improvement and evolution of our domain.

The column has two main goals. First, we will promote dialogue regarding topics of interests for the MM community, by providing tools for continuous discussion among the members of the multimedia community. Every quarter, we will discuss (usually) one topic via online tools. Topics will include “What is Multimedia, and what is the role of the Multimedia community in science?”; “Diversity and minorities in the community”; “The ACM code of ethics”; etc.

Second, we will monitor and summarize on-going discussions, and spread their results within and outside the community. Every edition of this column will then summarize the discussion, highlighting popular and non-popular opinions, agreed action points and future work.

To foster the discussion, we set up an online discussion forum to which all members of the multimedia community (expertise and seniority mixed) can participate: the Facebook MM Community Discussion group (follow this link: https://www.facebook.com/groups/132278853988735/) . For every edition of the column, we will choose an initial set of topics of high relevance for the community. We will include, for example, topics that have been previously discussed at ACM meetings (e.g., the code of ethics), or in related events (e.g., Diversity at MM Women lunch), or popular off-line discussions among MM researchers (e.g., review processes, vision of the scientific community…). In the first 15 days of the quarter, the members of the community will choose one topic from this short-list via an online poll shared through the MM Facebook group. We will then select the topic that received the higher number of votes as the subject for the quarterly discussion.

Volunteers or selected members of the MM group will start the discussion via Facebook posts on the group page. The discussion will be then open for a period of a month. All members of the community can participate by replying to posts or by directly posting on the group page, describing their point of view on the subject while being concise and clear. During this period, we will monitor and moderate (when needed) the discussion. At the end of the month, we will summarise the discussion by describing its evolution, exposing major and minor opinions, outlining highlights and lowlights. A final text with the summary and some relevant discussion extracts will be prepared and will appear in the SIGMM Records and in the Facebook “MM Community page”: https://www.facebook.com/MM-Community-217668705388738/.

Hopefully, the community will benefit from this initiative by either reaching some consensus or by pointing out important topics that are not mature enough and require further exploration. In the long-term, we hope these process will make the community evolve through large consensus and bottom-up discussions.

Let’s contribute and foster research around topics of high interest for the community!

Xavi and Miriam

Xavier Almeda-PinedaDr. Xavier Alameda-Pineda (Xavi) is research scientist at INRIA. Xavi’s interdisciplinary background (Msc in Mathematics, Telecommunications and Computer Science) grounded him to pursue his PhD in Mathematics and Computer Science, and a further post-doc in the University of Trento. His research interests are signal processing, computer vision and machine learning for scene and behavior understanding using multimodal data. He is the winner of the best paper award of ACM MM 2015, the best student paper award at IEEE WASPAA 2015 and the best scientific paper award at IAPR ICPR 2016.

 

 

Mariam RediDr. Miriam Redi is a research scientist in the Social Dynamics team at Bell Labs Cambridge. Her research focuses on content-based social multimedia understanding and culture analytics. In particular, Miriam explores ways to automatically assess visual aesthetics, sentiment, and creativity and exploit the power of computer vision in the context of web, social media, and online communities. Previously, she was a postdoc in the Social Media group at Yahoo Labs Barcelona and a research scientist at Yahoo London. Miriam holds a PhD from the Multimedia group in EURECOM, Sophia Antipolis.