In the rapidly evolving digital landscape, platforms like Criminal IP and Shodan are gaining attention for their ability to identify and respond to network and system vulnerabilities proactively. Criminal IP is a cyber threat intelligence platform that collects and analyzes information on exposed devices, services, and networks on the internet, detecting security vulnerabilities. It processes this data into a search engine format, allowing users to easily access the information they need. IoT platform Shodan offers similar capabilities, and both platforms have strengths in analyzing CVE (Common Vulnerabilities and Exposures). This article presents a comparative analysis of CVE data provided through the APIs of Criminal IP and Shodan.
Criminal IP vs. Shodan: The Process of Collecting Full CVE Data
CVE is a system for identifying security vulnerabilities found in specific software or systems in a standardized manner. It is an important criterion for security managers to accurately understand the vulnerabilities of their assets and prepare countermeasures. This article evaluates the number of each CVE findings and data reliability based on the search results from Criminal IP and Shodan platforms, for all CVEs reported from 1999 to the present. The goal is to help security administrators make optimal decisions.
For the analysis of Criminal IP vs. Shodan, all CVE data used was extracted and organized from the official databases. The detailed procedure for the collection process is as follows:
1. Source of All Published CVE Data
CVE data was obtained from the data feeds provided by the National Vulnerability Database (NVD). NVD is a trusted source that offers global data of common vulnerabilities and exposures (CVEs) in a standardized format.
2. Scope of Included CVE Data
NVD officially began providing CVE data in 2002, but the data also includes historical vulnerabilities from as early as 1999. This analysis, therefore, considers all data from 1999 to 2024. All CVEs published during this period were compared with detectable vulnerabilities in Criminal IP and Shodan.
3. CVE Data Updates and Reference Point
CVE data is updated daily, with new vulnerabilities continuously being added. This analysis is based on the latest data as of November 2024, and the reference point is as per the date when the data was arranged and analyzed.
4. Extracted CVE Data Entry for Criminal IP and Shodan Analysis
For each CVE ID, the following details were extracted:
- Modified date: The date the CVE was last modified, used to track the current state of the vulnerability data.
- Published date: The date the CVE was first published, indicating when the vulnerability was first identified.
In this article, the data above was used to check both the latest status of each CVE and the date of its initial publication, serving as a basis for analysis. The original CVE data provided by NVD is shown below.다.
API Used for CVE Data Validation of Criminal IP and Shodan
The API used in Criminal IP extracted CVE data on the combination of /v1/banner/search?query= with the cve_id filter. The returned results were in the format of ‘cve_id: CVE-2012-1234’. (More filters and their usage can be found on the Filters page.)
https://api.criminalip.io/v1/banner/search?query=cve_id:CVE-2021-1234
API used in Shodan is as follows. The query /shodan/host/search and the vuln: filter was used to search for the existence of CVE.
https://api.shodan.io/shodan/host/search?key={API_KEY}&query==vuln:CVE-2021-1234
When an IP with the vulnerability is detected through the API call above, both Criminal IP and Shodan return a JSON value similar to the one below. The values were counted and plotted as a graph.
Criminal IP vs. Shodan: Initial CVE Detection Results and Data Reliability
Despite Higher Number of Detection, Shodan Shows Lower CVE Data Reliability
The figure above shows the number of CVE vulnerabilities found in Criminal IP and Shodan by year. The blue graph is represents the number of CVE vulnerabilities detected by Criminal IP, and the red graph shows the number of CVEs detected by Shodan. The pattern is similar until around 2015, after when Shodan shows a higher number of detected CVEs. One thing to note, however, is the potential inclusion of data noise in the CVE system. In one IP case in Shodan illustrated below, for instance, NetData banner is shown displaying the result that relevant CVE exists. In this case, the CVE data should also display a vulnerability to NetData.
However, Shodan’s data displays the results as a CVE relevant to MySQL, as shown below. The number of CVEs is also returning an excessive number of results, as opposed to the actual CVE for MySQL. The following is an example of data with low reliability.
The case below reports that port 80 has a CVE vulnerability on the Apache server, but the CVE data displays the detail of CVE on MySQL, which is irrelevant to Apache. Similarly, a baseless trust on the CVE data from a mass of security platforms can lead to trust issues.
Criminal IP vs. Shodan: Real CVE Detections After Data Noise Cleansing
Both Criminal IP and Shodan may contain such discrepancies. The following method was employed with Criminal IP and Shodan APIs to refine the data:
- Before calling for CVEs individually, extract the product name (e.g., MySQL) and version, corresponding to the official CVE.
- Extract the real IP address with CVE using the cve_id and vuln: filters from Criminal IP and Shodan, respectively.
- Extract the product name and version from the packet of the IP address that was just extracted.
- If the extracted product name and version in step 1 match the data from the official CVE in step 3, the data is considered accurate and counted as an actual CVE. If it does not match, it is classified as a data error.
After applying this custom validation logic with an additional API, the year-by-year CVE detection graph for Criminal IP and Shodan was redrawn. Blue graph represents CVE detections of Criminal IP, and red shows that of Shodan’s. The statistics based on the initial analysis showed higher number of CVE detection in Shodan; after removing data noise, however, Criminal IP reveals higher number in CVE detection.
Summary and Insights
Security administrators must understand the characteristics of CVE detection systems and carefully compare data from each platform upon its use. Platforms like Criminal IP and Shodan, which seem similar on the surface, offer extensive data detection and quick scanning capabilities. Some of their results, however, as illustrated in the case of Shodan’s, may include vulnerabilities that are not related to the actual service. It is, therefore, essential to cross-verify the product name and version information related to the service banner to ensure the accuracy of the data.
The initial statistics showed that Shodan detected more CVEs. After filtering the data noise, however, Criminal IP showed a higher number of detected CVEs. The blue graph shows the number of CVE detections of Criminal IP, and the red graph shows the number of Shodan’s detections. The final results visually confirmed that Criminal IP had a higher data reliability.
What we can learn from this is that rather than blindly trusting the data provided by the API, the process of cross-validating the data and judging its accuracy based on various functions and combinations of APIs is fundamental. Instead of simply accepting the detected results, additional filtering and verification should be done to denoise the data and select information that can be actually be used. Making the most of the advantages of Criminal IP in this way allows for more effective and accurate security management.
Moreover, while CVEs have been the primary focus for vulnerability assessments, the U.S. Cybersecurity and Infrastructure Security Agency (CISA) is now highlighting KEVs (Known Exploit Vulnerabilities). Unlike CVEs, KEV only includes vulnerabilities that have been confirmed to have been used by attackers to perform malicious actions. Therefore, it is more accurate and covers only the vulnerabilities that deserve attention in the first place. Criminal IP plans to also provide KEV information in the future. The inclusion of this data is expected to enable more effective consideration of the priority and importance of vulnerabilities in the future.
About the Author
Young Yoon
Young Yoon, currently the CEO of ExploitWareLabs, is a leading expert in OSINT and penetration testing in South Korea. He has worked with more than hundreds of institutions and companies on web, infrastructure, and mobile app penetration testing projects, with his thorough expertise from A3 Security, KEPCO KDN, Secureone, and more. With over 20 years of experience in threat intelligence, attack surface management (ASM), dark web monitoring, and cyber threat hunting, Yoon has conducted numerous lectures and sessions. He has also served as a lecturer at the Korea Information Security Education Center (KISEC) and as an advisory member of the Cyber Security Hacking Organization Research Group of the National Police Agency, and the Hacking Research Group of the National Investigation Headquarters. He is actively supporting technical collaboration with various law enforcement agencies to prevent cybercrime.
Source: https://www.dailysecu.com/news/articleView.html?idxno=161689, https://nvd.nist.gov/vuln/data-feeds
Related Article: