Evgenii Zheltonozhskii

selected publications

1. arXiv

   Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels

   Evgenii Zheltonozhskii, Chaim Baskin, Avi Mendelson, Alex M. Bronstein, and Or Litany

   arXiv pre-print 2021

   Abs arXiv Bib Code

   The success of learning with noisy labels (LNL) methods relies heavily on the success of a warm-up stage where standard supervised training is performed using the full (noisy) training set. In this paper, we identify a "warm-up obstacle": the inability of standard warm-up stages to train high quality feature extractors and avert memorization of noisy labels. We propose "Contrast to Divide" (C2D), a simple framework that solves this problem by pre-training the feature extractor in a self-supervised fashion. Using self-supervised pre-training boosts the performance of existing LNL approaches by drastically reducing the warm-up stage’s susceptibility to noise level, shortening its duration, and increasing extracted feature quality. C2D works out of the box with existing methods and demonstrates markedly improved performance, especially in the high noise regime, where we get a boost of more than 27% for CIFAR-100 with 90% noise over the previous state of the art. In real-life noise settings, C2D trained on mini-WebVision outperforms previous works both in WebVision and ImageNet validation sets by 3% top-1 accuracy. We perform an in-depth analysis of the framework, including investigating the performance of different pre-training approaches and estimating the effective upper bound of the LNL performance with semi-supervised learning. Code for reproducing our experiments is available at this https URL

   @article{zheltonozhskii2021c2d,
     title = {Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels},
     author = {Zheltonozhskii, Evgenii and Baskin, Chaim and Mendelson, Avi and Bronstein, Alex M. and Litany, Or},
     journal = {arXiv pre-print},
     year = {2021},
     month = mar,
     url = {https://arxiv.org/abs/2103.13646},
     code = {https://github.com/ContrastToDivide/C2D},
     arxiv = {2103.13646},
     sortkey = {14},
     abbr = {arXiv},
     selected = {true},
     bibtex_show = {true}
   }
   

2. NeurIPS
   workshop

   Self-Supervised Learning for Large-Scale Unsupervised Image Clustering

   Evgenii Zheltonozhskii, Chaim Baskin, Alex M. Bronstein, and Avi Mendelson

   NeurIPS Self-Supervised Learning Workshop 2020

   Abs arXiv Bib Code

   Unsupervised learning has always been appealing to machine learning researchers and practitioners, allowing them to avoid an expensive and complicated process of labeling the data. However, unsupervised learning of complex data is challenging, and even the best approaches show much weaker performance than their supervised counterparts. Self-supervised deep learning has become a strong instrument for representation learning in computer vision. However, those methods have not been evaluated in a fully unsupervised setting. In this paper, we propose a simple scheme for unsupervised classification based on self-supervised representations. We evaluate the proposed approach with several recent self-supervised methods showing that it achieves competitive results for ImageNet classification (39% accuracy on ImageNet with 1000 clusters and 46% with overclustering). We suggest adding the unsupervised evaluation to a set of standard benchmarks for self-supervised learning. The code is available at this https URL

   @article{zheltonozhskii2020selfsupervised,
     title = {Self-Supervised Learning for Large-Scale Unsupervised Image Clustering},
     author = {Zheltonozhskii, Evgenii and Baskin, Chaim and Bronstein, Alex M. and Mendelson, Avi},
     journal = {NeurIPS Self-Supervised Learning Workshop},
     year = {2020},
     month = aug,
     url = {https://arxiv.org/abs/2008.10312},
     code = {https://github.com/Randl/kmeans_selfsuper},
     arxiv = {2008.10312},
     sortkey = {11},
     abbr = {NeurIPS <br> workshop},
     selected = {true},
     bibtex_show = {true}
   }
   

3. arXiv

   CAT: Compression-Aware Training for Bandwidth Reduction

   Chaim Baskin, Brian Chmiel, Evgenii Zheltonozhskii, Ron Banner, Alex M. Bronstein, and Avi Mendelson

   arXiv pre-print 2019

   Abs arXiv Bib Code

   Convolutional neural networks (CNNs) have become the dominant neural network architecture for solving visual processing tasks. One of the major obstacles hindering the ubiquitous use of CNNs for inference is their relatively high memory bandwidth requirements, which can be a main energy consumer and throughput bottleneck in hardware accelerators. Accordingly, an efficient feature map compression method can result in substantial performance gains. Inspired by quantization-aware training approaches, we propose a compression-aware training (CAT) method that involves training the model in a way that allows better compression of feature maps during inference. Our method trains the model to achieve low-entropy feature maps, which enables efficient compression at inference time using classical transform coding methods. CAT significantly improves the state-of-the-art results reported for quantization. For example, on ResNet-34 we achieve 73.1% accuracy (0.2% degradation from the baseline) with an average representation of only 1.79 bits per value. Reference implementation accompanies the paper at this https URL

   @article{baskin2019cat,
     title = {{CAT}: Compression-Aware Training for Bandwidth Reduction},
     author = {Baskin, Chaim and Chmiel, Brian and Zheltonozhskii, Evgenii and Banner, Ron and Bronstein, Alex M. and Mendelson, Avi},
     journal = {arXiv pre-print},
     year = {2019},
     month = sep,
     url = {https://arxiv.org/abs/1909.11481},
     code = {https://github.com/CAT-teams/CAT},
     arxiv = {1909.11481},
     sortkey = {6},
     abbr = {arXiv},
     selected = {true},
     bibtex_show = {true}
   }