ICCV 2017 Half-Day Tutorial on Adversarial Pattern Recognition


“If you know the enemy and know yourself, you need not fear the result of a hundred battles”
Sun Tzu, The art of war, 500 BC

 

 

Proposers' names, titles, affiliations, and emails:
•    Battista Biggio, IAPR Member, IEEE Senior Member;
•    Fabio Roli, IAPR Fellow, IEEE Fellow.
PRA Lab and Pluribus One, Italy
Email: battista.biggio (at) diee.unica.it
Email: surname_of_fabio (at) diee.unica.it

Material:  Slides - Web demo on Evasion Attacks 

Last update 27/11/2017: Video of the tutorial available

 

Tutorial description: Learning-based pattern classifiers, including deep networks, are currently used in several applications, ranging from computer vision to computer security. Most of these applications, including spam and malware detection, differ from traditional machine learning tasks, as the machine-learning algorithm has to face an intelligent and adaptive attacker who can carefully manipulate data with the goal of purposely subverting the learning process. Traditional machine learning techniques do not take into account the adversarial nature of classification problems like the ones mentioned above. One of the consequences is that the performance of standard pattern classifiers can significantly degrade when they are used in adversarial tasks. Pattern classifiers can be significantly vulnerable to well-crafted, sophisticated attacks exploiting knowledge of the learning and classification algorithms. Being increasingly adopted for security and privacy tasks, pattern recognition techniques will be soon targeted by specific attacks, crafted by skilled attackers. In particular, two main threats against learning algorithms have been identified, among a larger number of potential attack scenarios, respectively referred to as evasion and poisoning attacks [1-12].

Evasion attacks consist of manipulating input data at test time to cause misclassifications. These include, for instance, manipulation of malware code to have the corresponding sample undetected (i.e., misclassified as legitimate) [2,6]. Another instance is related to the existence of adversarial examples in computer-vision problems, namely, images that can be misclassified by deep-learning algorithms while being only imperceptibly distorted [11,12]. For instance, see the digit manipulations performed below through a straightforward gradient-descent evasion attack against an SVM classifier [2]. Accordingly, from a practical perspective, evasion attacks are already a relevant threat in several real-world application settings.

    

Poisoning attacks are subtler; their goal is to mislead the learning algorithm during the training phase by manipulating only a small fraction of the training data, in order to significantly increase the number of misclassified samples at test time, causing a denial of service [3,4,10]. These attacks require access to the training data used to learn the classification algorithm, which is possible in some application-specific contexts; for instance, in the case of systems which are re-trained or updated online, using data collected during system operation. Another category of systems that may be subject to poisoning attacks include those systems that exploit feedback from the end users to validate their decisions on some submitted samples, and then update the classification model accordingly (e.g., PDFRate, an online tool for detecting malware in PDF files [9]). Some examples of manipulated digit images capable of poisoning SVMs are given below. They are obtained with the gradient-based attack discussed in [3].

The problem of countering these threats and learning secure classifiers in adversarial settings is the subject of an emerging research field in the area of machine learning called adversarial machine learning.

The purposes of this tutorial are:

  • to introduce the fundamentals of adversarial machine learning to the computer vision community;
  • to illustrate the design cycle of a learning-based pattern recognition system for adversarial tasks;
  • to present novel techniques that have been recently proposed to assess performance of pattern classifiers under attack, evaluate classifiers’ vulnerabilities, and implement defence strategies that make learning algorithms and pattern classifiers more robust against attacks;
  • to show some applications of adversarial machine learning to pattern recognition tasks like object recognition in images, biometric identity recognition, spam and malware detection.


Structure of the tutorial

  1. Introduction to adversarial machine learning. Introduction by practical examples from computer vision, biometrics, spam, malware and network intrusion detection. Previous work on adversarial learning and recognition. Basic concepts and terminology. The concept of adversary-aware classifier. Definitions of attack and defence.
  2. Design of learning-based pattern classifiers in adversarial environments. Modelling adversarial tasks. The two-player model (the attacker and the classifier). Levels of reciprocal knowledge of the two players (perfect knowledge, limited knowledge, knowledge by queries and feedback). The concepts of security by design and security by obscurity.
  3. System design: vulnerability assessment and defence strategies. Attack models against pattern classifiers. The Influence of attacks on the classifier: causative or exploratory attacks. Type of security violation: integrity or availability attacks. The specificity of the attack: targeted or indiscriminate attacks. Vulnerability assessment by performance evaluation. Taxonomy of possible defence strategies. Examples from computer vision, biometrics, spam, malware and network intrusion detection.
  4. Summary and outlook. Current state of this research field and future perspectives.


Description of the target audience of the tutorial
This tutorial is devoted to:
(i) people who want to become aware of the new research field of adversarial machine learning and learn the fundamentals;
(ii) people doing research in machine learning and pattern recognition applications which have a potential adversarial component, and wish to learn how the techniques of adversarial classification can be effectively used in such applications.
No knowledge of the tutorial topics is assumed. A basic knowledge of machine learning and statistical pattern classification is requested.


References

  1. Barreno, M., Nelson, B., Sears, R., Joseph, A. D., Tygar, J. D. Can machine learning be secure? In ASIACCS ’06, pp. 16–25, 2006. ACM.
  2. Biggio, B., Corona, I., Maiorca, D., Nelson, B., Srndic, N., Laskov, P., Giacinto, G., Roli, F. Evasion attacks against machine learning at test time. In ECML-PKDD, Part III, vol. 8190 of LNCS, pp. 387– 402. Springer Berlin Heidelberg, 2013.
  3. Biggio, B., Nelson, B., Laskov, P. Poisoning attacks against SVMs. In Langford, J. and Pineau, J. (eds.), 29th ICML, pp. 1807–1814. Omnipress, 2012.
  4. Biggio, B., Fumera, G., Roli, F. Security evaluation of pattern classifiers under attack. IEEE Trans. Knowl. Data Eng., 26 (4):984–996, 2014.
  5. Huang, L., Joseph, A. D., Nelson, B., Rubinstein, B., Tygar, J. D. Adversarial machine learning. In AISec ’11, pp. 43–57, 2011.
  6. Maiorca, D., Corona, I., Giacinto, G. Looking at the bag is not enough to find the bomb: an evasion of structural methods for malicious PDF files detection. In ASIA CCS ’13, pp. 119–130, USA, 2013. ACM.
  7. Rish, I., Grabarnik, G. Sparse Modeling: Theory, Algorithms, and Applications. CRC Press, Inc., 2014.
  8. Rubinstein, B. I. P., Nelson, B., Huang, L., Joseph, A. D., Lau, S., Rao, S., Taft, N., Tygar, J. D. Antidote: understanding and defending against poisoning of anomaly detectors. In 9th IMC ’09, pp. 1–14, 2009. ACM.
  9. Smutz, C. and Stavrou, A. Malicious PDF detection using metadata and structural features. In 28th ACSAC ’12, pp. 239–248, 2012. ACM.
  10. Xiao, H., Biggio, B., Brown, G., Fumera, G., Eckert, C., Roli, F. Is feature selection secure against training data poisoning? In Bach, F. and Blei, D. (eds.), 32nd ICML, vol. 37, pp. 1689–1698, 2015.
  11. C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, R. Fergus. Intriguing properties of neural networks. In Int’l Conf. Learn. Repr., 2014.
  12. A. Bendale, T. Boult. Towards Open Set Deep Networks. In CVPR 2016.