With the advent of ubiquitous computing and Internet of Things (IoT), potentially billions of devices will create a broad range of data services and applications. Modern communication networks need to be designed to efficiently manage the increasing complexity. Cognitive network has been envisioned as a new paradigm to address this challenge, which has the capability of reasoning, planning and learning by incorporating cutting edge technologies including knowledge representation, context awareness, network optimization and machine learning. Cognitive network spans over the entire communication system including the core network and wireless links across the entire protocol stack. Cognitive Radio Network is a part of cognitive network over wireless links, which endeavors to better utilize the spectrum resources. However, the cognitive network communications have attracted various security threats, which become increasingly severe in pace with the growing complexity and adversity of the modern Internet. In this talk, I will focus on presenting a holistic multi-layer framework to protect network communications empowered by machine learning and data analytics. First, I will give an overview of the emerging security issues in cognitive network communications. Second, as no security mechanisms can be 100% successful in preventing all potential threats from entering a network, I will introduce an additional line of defense using network and behavior monitoring. I will present a systematic passive monitoring framework based on unsupervised machine learning methods to strategically monitor the network traffic and operations in order to detect abnormal and malicious network behaviors. Then, I will elaborate the design of data-driven botnet detection system, which employs cognitive technologies to protect the network communications from Peer-to-Peer (P2P) botnet threats. Finally, I will briefly discuss important research topics related to wireless, mobile communications and advanced botnet detection.