Despite their growing popularity, graph neural networks (GNNs) still have multiple unsolved problems, including finding more expressive aggregation methods, propagation of information to distant nodes, and training on large-scale graphs. Understanding and solving such problems require developing analytic tools and techniques. In this work, we propose the notion of recoverability, which is tightly related to information aggregation in GNNs, and based on this concept, develop the method for GNN embedding analysis. We define recoverability theoretically and propose a method for its efficient empirical estimation. We demonstrate, through extensive experimental results on various datasets and different GNN architectures, that estimated recoverability correlates with aggregation method expressivity and graph sparsification quality. Therefore, we believe that the proposed method could provide an essential tool for understanding the roots of the aforementioned problems, and potentially lead to a GNN design that overcomes them. The code to reproduce our experiments is available at this https URL
@article{fishman2022recoverability,title={On Recoverability of Graph Neural Network Representations},author={Fishman, Maxim and Baskin, Chaim and Zheltonozhskii, Evgenii and Banner, Ron and Mendelson, Avi},journal={arXiv pre-print},year={2022},month=jan,url={https://arxiv.org/abs/2201.12843},code={https://github.com/Anonymous1252022/Recoverability},arxiv={2201.12843},sortkey={16},abbr={arXiv},bibtex_show={true}}
CVPR’22
End-to-End Referring Video Object Segmentation with Multimodal Transformers
Adam Botach,
Evgenii Zheltonozhskii, and Chaim Baskin
In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Jun 2022
The referring video object segmentation task (RVOS) involves segmentation of a text-referred object instance in the frames of a given video. Due to the complex nature of this multimodal task, which combines text reasoning, video understanding, instance segmentation and tracking, existing approaches typically rely on sophisticated pipelines in order to tackle it. In this paper, we propose a simple Transformer-based approach to RVOS. Our framework, termed Multimodal Tracking Transformer (MTTR), models the RVOS task as a sequence prediction problem. Following recent advancements in computer vision and natural language processing, MTTR is based on the realization that video and text can both be processed together effectively and elegantly by a single multimodal Transformer model. MTTR is end-to-end trainable, free of text-related inductive bias components and requires no additional mask-refinement post-processing steps. As such, it simplifies the RVOS pipeline considerably compared to existing methods. Evaluation on standard benchmarks reveals that MTTR significantly outperforms previous art across multiple metrics. In particular, MTTR shows impressive +5.7 and +5.0 mAP gains on the A2D-Sentences and JHMDB-Sentences datasets respectively, while processing 76 frames per second. In addition, we report strong results on the public validation set of Refer-YouTube-VOS, a more challenging RVOS dataset that has yet to receive the attention of researchers. The code to reproduce our experiments is available at this https URL
@inproceedings{botach2021mttr,title={End-to-End Referring Video Object Segmentation with Multimodal Transformers},author={Botach, Adam and Zheltonozhskii, Evgenii and Baskin, Chaim},booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},year={2022},month=jun,url={https://arxiv.org/abs/2111.14821},code={https://github.com/mttr2021/MTTR},arxiv={2111.14821},sortkey={15},abbr={CVPR'22},selected={true},bibtex_show={true}}
WACV’22
Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels
The success of learning with noisy labels (LNL) methods relies heavily on the success of a warm-up stage where standard supervised training is performed using the full (noisy) training set. In this paper, we identify a "warm-up obstacle": the inability of standard warm-up stages to train high quality feature extractors and avert memorization of noisy labels. We propose "Contrast to Divide" (C2D), a simple framework that solves this problem by pre-training the feature extractor in a self-supervised fashion. Using self-supervised pre-training boosts the performance of existing LNL approaches by drastically reducing the warm-up stage’s susceptibility to noise level, shortening its duration, and increasing extracted feature quality. C2D works out of the box with existing methods and demonstrates markedly improved performance, especially in the high noise regime, where we get a boost of more than 27% for CIFAR-100 with 90% noise over the previous state of the art. In real-life noise settings, C2D trained on mini-WebVision outperforms previous works both in WebVision and ImageNet validation sets by 3% top-1 accuracy. We perform an in-depth analysis of the framework, including investigating the performance of different pre-training approaches and estimating the effective upper bound of the LNL performance with semi-supervised learning. Code for reproducing our experiments is available at this https URL
@inproceedings{zheltonozhskii2021c2d,title={Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels},author={Zheltonozhskii, Evgenii and Baskin, Chaim and Mendelson, Avi and Bronstein, Alex M. and Litany, Or},booktitle={IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},month=jan,year={2022},pages={1657-1667},url={https://openaccess.thecvf.com/content/WACV2022/html/Zheltonozhskii_Contrast_To_Divide_Self-Supervised_Pre-Training_for_Learning_With_Noisy_Labels_WACV_2022_paper.html},code={https://github.com/ContrastToDivide/C2D},arxiv={2103.13646},doi={10.1109/WACV51458.2022.00046},sortkey={14},abbr={WACV'22},video={https://www.youtube.com/watch?v=0ta4RVqh0vM},poster={https://video.vast.uccs.edu/WACV22/WACV/580-wacv-poster.pdf},selected={true},bibtex_show={true}}
We consider the problem of extracting semantic attributes, using only classification labels for supervision. For example, when learning to classify images of birds into species, we would like to observe the emergence of features used by zoologists to classify birds. To tackle this problem, we propose training a neural network with discrete features in the last layer, followed by two heads: a multi-layered perceptron (MLP) and a decision tree. The decision tree utilizes simple binary decision stumps, thus encouraging features to have semantic meaning. We present a theoretical analysis, as well as a practical method for learning in the intersection of two hypothesis classes. Compared with various benchmarks, our results show an improved ability to extract a set of features highly correlated with a ground truth set of unseen attributes.
@inproceedings{ali2021semantic,title={Weakly Supervised Recovery of Semantic Attributes},author={Ali, Ameen and Galanti, Tomer and Zheltonozhskii, Evgenii and Baskin, Chaim and Wolf, Lior},booktitle={First Conference on Causal Learning and Reasoning},year={2022},month=apr,url={https://openreview.net/forum?id=GdAzRedTV7J},arxiv={2103.11888},sortkey={13},abbr={CLeaR'22},bibtex_show={true}}
The demand for running NNs in embedded environments has increased significantly in recent years due to the significant success of convolutional neural network (CNN) approaches in various tasks, including image recognition and generation. The task of achieving high accuracy on resource-restricted devices, however, is still considered to be challenging, which is mainly due to the vast number of design parameters that need to be balanced. While the quantization of CNN parameters leads to a reduction of power and area, it can also generate unexpected changes in the balance between communication and computation. This change is hard to evaluate, and the lack of balance may lead to lower utilization of either memory bandwidth or computational resources, thereby reducing performance. This paper introduces a hardware performance analysis framework for identifying bottlenecks in the early stages of CNN hardware design. We demonstrate how the proposed method can help in evaluating different architecture alternatives of resource-restricted CNN accelerators (e.g., part of real-time embedded systems) early in design stages and, thus, prevent making design mistakes.
@article{karbachevsky2021earlystage,title={Early-Stage Neural Network Hardware Performance Analysis},author={Karbachevsky, Alex and Baskin, Chaim and Zheltonozhskii, Evgenii and Yermolin, Yevgeny and Gabbay, Freddy and Bronstein, Alex M. and Mendelson, Avi},journal={Sustainability},issuetitle={Energy-Efficient Computing Systems for Deep Learning},editor={Cano, José and Abellán, José L. and Kaeli, David},volume={13},number={2},issn={2071-1050},publisher={MDPI AG},pages={717},year={2021},month=jan,url={http://dx.doi.org/10.3390/su13020717},html={https://www.mdpi.com/2071-1050/13/2/717},doi={10.3390/su13020717},arxiv={2004.08906},sortkey={10},abbr={Sustainability},bibtex_show={true}}
Neural network quantization enables the deployment of large models on resource-constrained devices. Current post-training quantization methods fall short in terms of accuracy for INT4 (or lower) but provide reasonable accuracy for INT8 (or above). In this work, we study the effect of quantization on the structure of the loss landscape. We show that the structure is flat and separable for mild quantization, enabling straightforward post-training quantization methods to achieve good results. We show that with more aggressive quantization, the loss landscape becomes highly non-separable with steep curvature, making the selection of quantization parameters more challenging. Armed with this understanding, we design a method that quantizes the layer parameters jointly, enabling significant accuracy improvement over current post-training quantization methods. Reference implementation is available at https://github.com/ynahshan/nn-quantization-pytorch/tree/master/lapq.
@article{nahshan2019lapq,title={Loss Aware Post-Training Quantization},author={Nahshan, Yury and Chmiel, Brian and Baskin, Chaim and Zheltonozhskii, Evgenii and Banner, Ron and Bronstein, Alex M. and Mendelson, Avi},journal={Machine Learning},year={2021},month=oct,code={https://github.com/ynahshan/nn-quantization-pytorch/tree/master/lapq},doi={10.1007/s10994-021-06053-z},arxiv={1911.07190},issn={1573-0565},sortkey={7},abbr={ML},bibtex_show={true},url={https://link.springer.com/article/10.1007/s10994-021-06053-z}}
One major obstacle hindering the ubiquitous use of CNNs for inference is their relatively high memory bandwidth requirements, which can be the primary energy consumer and throughput bottleneck in hardware accelerators. Inspired by quantization-aware training approaches, we propose a compression-aware training (CAT) method that involves training the model to allow better compression of weights and feature maps during neural network deployment. Our method trains the model to achieve low-entropy feature maps, enabling efficient compression at inference time using classical transform coding methods. CAT significantly improves the state-of-the-art results reported for quantization evaluated on various vision and NLP tasks, such as image classification (ImageNet), image detection (Pascal VOC), sentiment analysis (CoLa), and textual entailment (MNLI). For example, on ResNet-18, we achieve near baseline ImageNet accuracy with an average representation of only 1.5 bits per value with 5-bit quantization. Moreover, we show that entropy reduction of weights and activations can be applied together, further improving bandwidth reduction. Reference implementation is available.
@article{baskin2019cat,title={{CAT}: Compression-Aware Training for Bandwidth Reduction},author={Baskin, Chaim and Chmiel, Brian and Zheltonozhskii, Evgenii and Banner, Ron and Bronstein, Alex M. and Mendelson, Avi},journal={Journal of Machine Learning Research},year={2021},month=aug,url={http://jmlr.org/papers/v22/20-1374.html},code={https://github.com/CAT-teams/CAT},arxiv={1909.11481},volume={22},number={269},pages={1-20},sortkey={6},abbr={JMLR},bibtex_show={true}}
Convolutional Neural Networks (CNNs) are very popular in many fields including computer vision, speech recognition, natural language processing, etc. Though deep learning leads to groundbreaking performance in those domains, the networks used are very computationally demanding and are far from being able to perform in real-time applications even on a GPU, which is not power efficient and therefore does not suit low power systems such as mobile devices. To overcome this challenge, some solutions have been proposed for quantizing the weights and activations of these networks, which accelerate the runtime significantly. Yet, this acceleration comes at the cost of a larger error unless spatial adjustments are carried out. The method proposed in this work trains quantized neural networks by noise injection and a learned clamping, which improve accuracy. This leads to state-of-the-art results on various regression and classification tasks, e.g., ImageNet classification with architectures such as ResNet-18/34/50 with as low as 3 bit weights and activations. We implement the proposed solution on an FPGA to demonstrate its applicability for low-power real-time applications. The quantization code will become publicly available upon acceptance.
@article{baskin2018nice,title={{NICE}: Noise Injection and Clamping Estimation for Neural Network Quantization},author={Baskin, Chaim and Zheltonozhskii, Evgenii and Rozen, Tal and Liss, Natan and Chai, Yoav and Schwartz, Eli and Giryes, Raja and Bronstein, Alexander M. and Mendelson, Avi},journal={Mathematics},issuetitle={Computational Optimizations for Machine Learning},editor={Gabbay, Freddy},volume={9},number={17},issn={2227-7390},publisher={MDPI AG},year={2021},month=sep,url={https://www.mdpi.com/2227-7390/9/17/2144},doi={10.3390/math9172144},arxiv={1810.00162},sortkey={3},abbr={Mathematics},bibtex_show={true}}
We present a novel method for neural network quantization. Our method, named UNIQ, emulates a non-uniform k-quantile quantizer and adapts the model to perform well with quantized weights by injecting noise to the weights at training time. As a by-product of injecting noise to weights, we find that activations can also be quantized to as low as 8-bit with only a minor accuracy degradation. Our non-uniform quantization approach provides a novel alternative to the existing uniform quantization techniques for neural networks. We further propose a novel complexity metric of number of bit operations performed (BOPs), and we show that this metric has a linear relation with logic utilization and power. We suggest evaluating the trade-off of accuracy vs. complexity (BOPs). The proposed method, when evaluated on ResNet18/34/50 and MobileNet on ImageNet, outperforms the prior state of the art both in the low-complexity regime and the high accuracy regime. We demonstrate the practical applicability of this approach, by implementing our non-uniformly quantized CNN on FPGA.
@article{baskin2018uniq,title={{UNIQ:} Uniform Noise Injection for Non-Uniform Quantization of Neural Networks},author={Baskin, Chaim and Liss, Natan and Schwartz, Eli and Zheltonozhskii, Evgenii and Giryes, Raja and Bronstein, Alex M. and Mendelson, Avi},journal={ACM Transactions on Computer Systems},volume={37},number={1–4},issn={0734-2071},publisher={Association for Computing Machinery},address={New York, NY, USA},issue_date={March 2021},articleno={4},numpages={15},year={2021},month=mar,url={https://arxiv.org/abs/1804.10969},doi={10.1145/3444943},arxiv={1804.10969},sortkey={2},abbr={ACM TOCS},bibtex_show={true}}
2020
arXiv
Single-Node Attack for Fooling Graph Neural Networks
Graph neural networks (GNNs) have shown broad applicability in a variety of domains. Some of these domains, such as social networks and product recommendations, are fertile ground for malicious users and behavior. In this paper, we show that GNNs are vulnerable to the extremely limited scenario of a single-node adversarial example, where the node cannot be picked by the attacker. That is, an attacker can force the GNN to classify any target node to a chosen label by only slightly perturbing another single arbitrary node in the graph, even when not being able to pick that specific attacker node. When the adversary is allowed to pick a specific attacker node, the attack is even more effective. We show that this attack is effective across various GNN types, such as GraphSAGE, GCN, GAT, and GIN, across a variety of real-world datasets, and as a targeted and a non-targeted attack. Our code is available at this https URL
@article{finkelshtein2020single,title={Single-Node Attack for Fooling Graph Neural Networks},author={Finkelshtein, Ben and Baskin, Chaim and Zheltonozhskii, Evgenii and Alon, Uri},journal={arXiv pre-print},year={2020},month=nov,url={https://arxiv.org/abs/2011.03574},code={https://github.com/benfinkelshtein/SINGLE},arxiv={2011.03574},sortkey={12},abbr={arXiv},bibtex_show={true}}
NeurIPS workshop
Self-Supervised Learning for Large-Scale Unsupervised Image Clustering
Unsupervised learning has always been appealing to machine learning researchers and practitioners, allowing them to avoid an expensive and complicated process of labeling the data. However, unsupervised learning of complex data is challenging, and even the best approaches show much weaker performance than their supervised counterparts. Self-supervised deep learning has become a strong instrument for representation learning in computer vision. However, those methods have not been evaluated in a fully unsupervised setting. In this paper, we propose a simple scheme for unsupervised classification based on self-supervised representations. We evaluate the proposed approach with several recent self-supervised methods showing that it achieves competitive results for ImageNet classification (39% accuracy on ImageNet with 1000 clusters and 46% with overclustering). We suggest adding the unsupervised evaluation to a set of standard benchmarks for self-supervised learning. The code is available at this https URL
@article{zheltonozhskii2020unsupervised,title={Self-Supervised Learning for Large-Scale Unsupervised Image Clustering},author={Zheltonozhskii, Evgenii and Baskin, Chaim and Bronstein, Alex M. and Mendelson, Avi},journal={NeurIPS Self-Supervised Learning Workshop},year={2020},month=aug,url={https://arxiv.org/abs/2008.10312},code={https://github.com/Randl/kmeans_selfsuper},arxiv={2008.10312},sortkey={11},abbr={NeurIPS <br> workshop},selected={true},bibtex_show={true}}
arXiv
Colored Noise Injection for Training Adversarially Robust Neural Networks
Even though deep learning has shown unmatched performance on various tasks, neural networks have been shown to be vulnerable to small adversarial perturbations of the input that lead to significant performance degradation. In this work we extend the idea of adding white Gaussian noise to the network weights and activations during adversarial training (PNI) to the injection of colored noise for defense against common white-box and black-box attacks. We show that our approach outperforms PNI and various previous approaches in terms of adversarial accuracy on CIFAR-10 and CIFAR-100 datasets. In addition, we provide an extensive ablation study of the proposed method justifying the chosen configurations.
@article{zheltonozhskii2020cni,title={Colored Noise Injection for Training Adversarially Robust Neural Networks},author={Zheltonozhskii, Evgenii and Baskin, Chaim and Nemcovsky, Yaniv and Chmiel, Brian and Mendelson, Avi and Bronstein, Alex M.},journal={arXiv pre-print},year={2020},month=mar,url={https://arxiv.org/abs/2003.02188},arxiv={2003.02188},sortkey={9},abbr={arXiv},bibtex_show={true}}
Convolutional neural networks (CNNs) achieve state-of-the-art accuracy in a variety of tasks in computer vision and beyond. One of the major obstacles hindering the ubiquitous use of CNNs for inference on low-power edge devices is their high computational complexity and memory bandwidth requirements. The latter often dominates the energy footprint on modern hardware. In this paper, we introduce a lossy transform coding approach, inspired by image and video compression, designed to reduce the memory bandwidth due to the storage of intermediate activation calculation results. Our method does not require fine-tuning the network weights and halves the data transfer volumes to the main memory by compressing feature maps, which are highly correlated, with variable length coding. Our method outperform previous approach in term of the number of bits per value with minor accuracy degradation on ResNet-34 and MobileNetV2. We analyze the performance of our approach on a variety of CNN architectures and demonstrate that FPGA implementation of ResNet-18 with our approach results in a reduction of around 40% in the memory energy footprint, compared to quantized network, with negligible impact on accuracy. When allowing accuracy degradation of up to 2%, the reduction of 60% is achieved. A reference implementation accompanies the paper.
@inproceedings{chmiel2020transformcoding,title={Feature Map Transform Coding for Energy-Efficient CNN Inference},author={Chmiel, Brian and Baskin, Chaim and Banner, Ron and Zheltonozhskii, Evgenii and Yermolin, Yevgeny and Karbachevsky, Alex and Bronstein, Alex M. and Mendelson, Avi},booktitle={International Joint Conference on Neural Networks (IJCNN)},pages={1-9},year={2020},month=jul,url={https://arxiv.org/abs/1905.10830},code={https://github.com/CompressTeam/TransformCodingInference},doi={10.1109/IJCNN48605.2020.9206968},arxiv={1905.10830},sortkey={5},abbr={IJCNN},award={Oral},bibtex_show={true}}
2019
arXiv
Smoothed Inference for Adversarially-Trained Models
Deep neural networks are known to be vulnerable to adversarial attacks. Current methods of defense from such attacks are based on either implicit or explicit regularization, e.g., adversarial training. Randomized smoothing, the averaging of the classifier outputs over a random distribution centered in the sample, has been shown to guarantee the performance of a classifier subject to bounded perturbations of the input. In this work, we study the application of randomized smoothing as a way to improve performance on unperturbed data as well as to increase robustness to adversarial attacks. The proposed technique can be applied on top of any existing adversarial defense, but works particularly well with the randomized approaches. We examine its performance on common white-box (PGD) and black-box (transfer and NAttack) attacks on CIFAR-10 and CIFAR-100, substantially outperforming previous art for most scenarios and comparable on others. For example, we achieve 60.4% accuracy under a PGD attack on CIFAR-10 using ResNet-20, outperforming previous art by 11.7%. Since our method is based on sampling, it lends itself well for trading-off between the model inference complexity and its performance. A reference implementation of the proposed techniques is provided at this https URL
@article{nemcovsky2019smoothed,title={Smoothed Inference for Adversarially-Trained Models},author={Nemcovsky, Yaniv and Zheltonozhskii, Evgenii and Baskin, Chaim and Chmiel, Brian and Bronstein, Alex M. and Mendelson, Avi},journal={arXiv pre-print},year={2019},month=nov,url={https://arxiv.org/abs/1911.07198},code={https://github.com/yanemcovsky/SIAM},arxiv={1911.07198},sortkey={8},abbr={arXiv},bibtex_show={true}}
ICML workshop
Towards Learning of Filter-Level Heterogeneous Compression of Convolutional Neural Networks
Recently, deep learning has become a de facto standard in machine learning with convolutional neural networks (CNNs) demonstrating spectacular success on a wide variety of tasks. However, CNNs are typically very demanding computationally at inference time. One of the ways to alleviate this burden on certain hardware platforms is quantization relying on the use of low-precision arithmetic representation for the weights and the activations. Another popular method is the pruning of the number of filters in each layer. While mainstream deep learning methods train the neural networks weights while keeping the network architecture fixed, the emerging neural architecture search (NAS) techniques make the latter also amenable to training. In this paper, we formulate optimal arithmetic bit length allocation and neural network pruning as a NAS problem, searching for the configurations satisfying a computational complexity budget while maximizing the accuracy. We use a differentiable search method based on the continuous relaxation of the search space proposed by Liu et al. (arXiv:1806.09055). We show, by grid search, that heterogeneous quantized networks suffer from a high variance which renders the benefit of the search questionable. For pruning, improvement over homogeneous cases is possible, but it is still challenging to find those configurations with the proposed method. The code is publicly available at this https URL and this https URL
@article{zur2019filterlevel,title={Towards Learning of Filter-Level Heterogeneous Compression of Convolutional Neural Networks},author={Zur, Yochai and Baskin, Chaim and Zheltonozhskii, Evgenii and Chmiel, Brian and Evron, Itay and Bronstein, Alex M. and Mendelson, Avi},journal={ICML AutoML Workshop},year={2019},month=apr,url={https://arxiv.org/abs/1904.09872},code={https://github.com/yochaiz/Slimmable},arxiv={1904.09872},sortkey={4},abbr={ICML <br> workshop},bibtex_show={true}}
2018
IPDPS workshop
Streaming Architecture for Large-Scale Quantized Neural Networks on an FPGA-Based Dataflow Platform
Deep neural networks (DNNs) are used by different applications that are executed on a range of computer architectures, from IoT devices to supercomputers. The footprint of these networks is huge as well as their computational and communication needs. In order to ease the pressure on resources, research indicates that in many cases a low precision representation (1-2 bit per parameter) of weights and other parameters can achieve similar accuracy while requiring less resources. Using quantized values enables the use of FPGAs to run NNs, since FPGAs are well fitted to these primitives; e.g., FPGAs provide efficient support for bitwise operations and can work with arbitrary-precision representation of numbers.
This paper presents a new streaming architecture for running QNNs on FPGAs. The proposed architecture scales out better than alternatives, allowing us to take advantage of systems with multiple FPGAs. We also included support for skip connections, that are used in state-of-the art NNs, and shown that our architecture allows to add those connections almost for free. All this allowed us to implement an 18-layer ResNet for 224x224 images classification, achieving 57.5% top-1 accuracy.
In addition, we implemented a full-sized quantized AlexNet. In contrast to previous works, we use 2-bit activations instead of 1-bit ones, which improves AlexNet’s top-1 accuracy from 41.8% to 51.03% for the ImageNet classification. Both AlexNet and ResNet can handle 1000-class real-time classification on an FPGA.
Our implementation of ResNet-18 consumes 5x less power and is 4x slower for ImageNet, when compared to the same NN on the latest Nvidia GPUs. Smaller NNs, that fit a single FPGA, are running faster then on GPUs on small (32x32) inputs, while consuming up to 20x less energy and power.
@inproceedings{baskin2018streaming,title={Streaming Architecture for Large-Scale Quantized Neural Networks on an FPGA-Based Dataflow Platform},author={Baskin, Chaim and Liss, Natan and Zheltonozhskii, Evgenii and Bronstein, Alex M. and Mendelson, Avi},booktitle={IEEE International Parallel and Distributed Processing Symposium Workshops},pages={162--169},year={2018},month=may,url={https://arxiv.org/abs/1708.00052},doi={10.1109/IPDPSW.2018.00032},arxiv={1708.00052},sortkey={1},abbr={IPDPS <br> workshop},bibtex_show={true}}