OSINT and Top 15 Open-source Intelligence Tools

• Asking questions on any search engine.
• Research public forums on the latest mobile technologies.
• Watch a YouTube video on how to make a certain delicacy.

Importance of OSINT

OSINT, in general, helps an organization keep tabs on public information. It also helps in reducing the potential attack surface and thus prevents breaches and leaks. For example, the following tasks are done with the help of OSINT.

Discovering and locating the assets outside the perimeter or public: OSINT helps IT and cybersecurity teams discover and locate public-facing assets. Through OSINT, the information available in each asset can then be mapped and assessed for sensitive or critical information that can be exploited. In general, the OSINT tools help in mapping and recording data of the public assets of the company that is publicly available and accessible.

 Finding relevant data and information outside the organization: The OSINT tools help find relevant data outside the organization, such as domains or ports outside the organization’s network perimeter. This function is particularly helpful for an organization that has recently merged or acquired another organization as it helps find relevant information available outside of the organization just acquired.

Take necessary measures with the collated data: The data collected can be massive and not in order. OSINT tools convert the data into meaningful information that can be used as actionable intelligence. OSINT tools also help piece the data together and deal with sensitive data and their problems on a priority basis.

Read Also: How to Protect Yourself From DDoS Attacks

OSINT Techniques

• What are you looking for?
• What is your main research objective?
• Who is your main target?
• What tools or mechanisms will you undertake to conduct the research?

 OSINT techniques can be divided into two major categories, namely Active OSINT and Passive OSINT

 Active OSINT: Includes port and system scanning and direct contact with the target. The results are more reliable and dependable, along with a high risk of detection.

Passive OSINT: In this category, contact is established with the help of third-party services. Since it includes a third party, the search results may not be reliable and may include many false positives and negatives. Therefore, the risk of detection is quite low in this category.

Read Also: What Is Zero Trust Security?

Challenges in Performing OSINT

Certain risks that are involved with OSINT tools are:

Getting detected: This is the most common risk involved as performing an OSINT investigation may give your information away as the one who was searching for the data.

Losing access to the information: Getting detected may result in you losing access to the information as it may lead to securing publicly available information or hiding the trails.

You become the victim: If your cover is blown, you can risk becoming a target of an investigation or, even worse, spying.

Content filtering: OSINT performs an exponential search and collates a humongous amount of data. If the data is not pieced together or in order, the data collected is useless and doesn’t result in any meaningful action.

Top OSINT Tools

To counter the challenges in performing OSINT, the following tools help with the collection of information and data:

BuiltWith: As the name suggests, BuiltWith lets you decode or find out what the websites are built with or made up of. It enables the user to identify different tech stacks and platforms that power the websites. For instance, BuiltWith can identify whether the website is built using Joomla, WordPress or Drupal as its CMS. It also identifies and generates a list of Java scripts/CSS libraries, website plugins, website framework and server information. BuiltWith can be used as preliminary research or an observation tool for websites.

Maltego: Maltego is primarily used for uncovering relationships among domains and publicly accessible information. It also helps in charting the humongous data into readable and easy-to-understand charts and graphs, which helps convert the raw data into some actionable measures. Maltego comes with 58 data integrations from over 35 data partners that allow users to choose four different layouts to recognize patterns in the data they’ve uncovered and piece complex data together.

Mitaka: Mitaka is available as a Chrome extension and as a Firefox browser adds on that helps in searching IP addresses, URLs, domains, hashes and wallet addresses across six dozen search engines. It also helps the cyber security team recognize and detect various indicators of compromise from your web browser and helps mitigate threats and risks. Additionally, as they are extensions, online databases can be quickly queried with just a click.

Spyse: Spyse is considered the complete internet assets registry and is used to collect the data on servers, websites and peripheral connected devices which are publicly available. This is also used as a reconnaissance tool that conducts data analysis to detect any security vulnerabilities of unmanaged assets and also helps in securing exposed credentials.

Spiderfoot: Spiderfoot is a free OSINT reconnaissance tool that integrates with different multiple data sources and automates the collection of OSINT. Spiderfoot gathers and analyses data regarding domains, IP addresses, CIDR ranges, phone numbers, usernames, and other sensitive data. Providing an intuitive web-based GUI, Spiderfoot contains both a command-line interface and an embedded web server, making it ideal for red team reconnaissance activities. In general, Spiderfoot helps discover more information about your target or identify what your organization may be inadvertently exposed to the public.

OSINT framework: The OSINT framework doesn’t run on servers but is a web-based interface that is useful in gaining valuable information and data by querying free search engines, resources and tools and helps you sniff out the data you need by breaking down the different topics of interest. The querying of websites for data extraction is free, while some require registration and have paid versions that help you construct an advanced google search and collect in-depth data.

Creepy: Creepy is a tool written in python that helps in collecting geolocation data of any individuals through a query raised through social networking platforms and hosting services. Creepy enables the user to present or plot the data collected on a map. It also allows the users to download or filter the data. CSV or .KML format.

Recon-ng: Recon-ng is a tool written in python that primarily focuses on web-based web-based open-source reconnaissance. It includes many modules, interactive help and convenience functions that guide users to use the tool correctly. Recon-ng automates activities like cutting pasting and harvesting, which are time-consuming and also performs operations like database interaction, performing web requests and managing API keys.

Shodan: Shodan is a type of network security monitor and a dedicated search engine used to find data and intelligence on the internet of things (IoT). This tool is also known as the search engine of hackers as it helps you find and explore different devices connected to a network. It also helps detect and find open ports and vulnerabilities on the attack surface. Shodan is of specific interest for IT professionals as it gives information and details about HTTP, SSP, SNMP, RTSP, which are based on operating systems, countries, networks and ports. Along with the IoT devices, Shodan can also query databases and find data publicly accessible through paths other than the main interface.

theHarvester: theHarvester is one of the simplest tools to capture and access public information outside an organization’s network perimeter. It brings back valuable information about virtual hosts, subdomain names, email addresses and open ports of any organization. This tool is very helpful in determining the scope of the pen test and helps as a reconnaissance step before pen testing. theHarvester uses popular search engines like Google, Duck Duck Go, Bing and social media networks to collect OSINT.

Metagoofil: As the name suggests, Metagoofil is used to extract metadata from public documents that also include PDFs and Microsoft office files. It finds the target document and stores it on a local disk and maps the paths used to get the documents. This helps obtain directory tree information, shared resources and server names of the host organization. This is a perfect tool for hackers to gather information and launch brute-force attacks on the target system. This tool also helps cyber security professionals determine the vulnerabilities and helps secure the network by closing the gaps before the hacker exploits these vulnerabilities.

Censys: This is a wonderful tool that acts as a search engine to get information about any device or network system connected to the internet. They can also return information on servers and domain names. In addition, you can find geoinformation and technical details about 80 and 443 ports running on a server, HTTP mapping of the target website, SSL certificate information, TLS handshake information, and WHOIS information.

TinEye: TInEye is an image search and image recognition tool that mainly focuses on reverse image searches that helps moderate the content that is posted on the web and is available for access through public domains. It can detect instances of fraud and copyright occurring through images pattern recognition and track the location of these images online among the constantly growing index of billion images online

OpenVAS: Open vulnerability assessment is a security framework that includes a vulnerability scanner for IT professionals to detect threats and vulnerabilities in a system. It is used for authenticated and unauthenticated testing, performance tuning for scans, high-level industrial level protocols and a powerful internal programming language to carry out vulnerability tests from a continuous daily feed. It enforces security by enabling continuous monitoring of networks systems and applications for threats and vulnerabilities. 

Searchcode: Searchcode is a unique and dedicated search engine that searches the code repository for any intelligence inside free source code. Works like any other normal search engine, but instead of searching for indexed web servers, searchcode searches for information in the code repositories of running apps or apps in the developing stage. It is completely free, and its filters make it easy for the users to sort data by language, repository or phrase. It is a good OSINT tool since it gathers information from accessible source codes and checks for sensitive information. Searchcode Is a good tool to have when the apps are in the developmental stage and can be used as a reconnaissance tool before the deployment stage.