In drive-by-download attacks, malware executable are downloaded and installed automatically by exploiting browser vulnerabilities. Botnets, e.g., Torpig, often use drive-by-download as the initial infection vector. In contrast, legitimate download activities are triggered by explicit user requests, e.g., a click on a "Save file as" button in a dialog window. We describe a host-based detection approach against drive-by-downloads by exploring the behaviors and knowledge of human-users regarding file systems. Specifically, we examine the use of (trusted) user inputs to mark the trustworthiness of file system properties. This solution involves capturing keyboard/mouse inputs of user, and correlating these input events to file-downloading events. We describe our mechanism for monitoring file-creation events and our design and implementation of a security framework for controlling the access of processes to file systems.