Open Source Intelligence (OSINT) 101

Open Source Intelligence, better known as OSINT, is a technology that refers publicly available and open sources of information (as opposed to covert or secret sources) used in connection with intelligence. OSINT is information that comes from public and open sources. A large amount of actionable and predictable intelligence is obtained from public and non-classified sources. It means that the information collected is available not only to the public for consumption, but also to the entire intelligence community.

The use of open-source intelligence dates back to World War II, but this process has been formally used since the time when the US government recognized the importance of using information from open sources such as radio broadcasts and the press. The Central Intelligence Agency (CIA) has established an Open Source Center to build on the established expertise of its predecessor, the Office of Strategic Services (OSS). The main objective was to understand, translate and analyze foreign radio broadcasts and the press, as well as to access information about foreign governments and organizations.

Open-source information is available to the public in the form of resources and websites that can be located and found via online search engines, which greatly facilitates the collection of open-source information. Open source is considered more accessible than traditional methods of gathering information, as it does not require specific techniques, tools, or skills to access the information.

This is why it is so important to use open-source intelligence. It gives you the ability to find and fix vulnerabilities in your organization’s network, fix them, and remove sensitive information before threat actors use the same tools and techniques to exploit them. Success is a key component of OSINT analysts because, without it, researchers can get into serious trouble by inadvertently alerting their targets, improperly collecting data, or making it less useful when delivered to customers.

While the definition of OSINT is becoming more and more complicated, people are increasingly making use of open-source intelligence in today’s world. There are a lot of people out there tasked with digging through the Internet and putting all the data and information they are willing to publish online.

The raw data is collected and analyzed to help you understand a little more clearly, but obtaining the information does not require any kind of covert effort. A good percentage of this data is publicly available, and knowing how to use it opens the door for organizations to expand their activities and operate more efficiently in today’s information age.

Data protection laws such as the GDPR have only strengthened the use of appropriate OSINT systems. All information collected during the search is available openly and without a search warrant or subpoena. The information can be retrieved in a legal and copyrighted manner.

Although similar intelligence technologies have existed for hundreds of years, OSINT has gained momentum in recent years due to the rise of the Internet and the proliferation of open sources.

The use of publicly available information to gather information carries risks compared to the use of human resources on the ground, particularly in hostile countries, or the use of spy satellites. OSINT can be used in a wide range of situations, such as cyber-attacks, cyber espionage, and cyber operations.

OSINT collecting is generally cheaper than other intelligence sources. While there are a number of open-source tools for use in the intelligence community, one of them is the search engine, Google, as most people call it. Conducting social engineering attacks on targets is also seen as a form of active information gathering.

One of the biggest problems security experts face is that users accidentally disclose sensitive assets and information on the Internet. This is why it is so important to use open source intelligence for security purposes, as it gives you the ability to find, fix, and remove sensitive information using the same tools and techniques that threat actors use to exploit it.

Whether you are looking for information about a terrorist attack, drug trafficking, or a criminal investigation, it is important to understand what open-source intelligence services really are and how they work including its collection, use, and use, as well as the tools and techniques that can be used to capture and analyze it.

Collecting relevant information for cybersecurity investigations is a difficult task, especially when operating with limited information about adversaries. OSINT is non-sensitive intelligence used by analysts to answer classified, non-classified, and proprietary intelligence requirements in previous intelligence disciplines.

OSINT process

>First, you need to have a good understanding of the process of sourcing and using open source intelligence, as well as your organization’s security policies and procedures.

OSINT resources and techniques

OSINT is extensively used by hackers and penetration testers to collect information about specific targets online. One of the biggest problems security experts face is the regularity with which normal, well, users accidentally publish sensitive assets and information on the Internet. OSINT to combat data leaks and to know what is causing future cyber threats, and to create an organization’s threat strategy by analyzing OSINT sources within the organization and combining this information with other information to achieve an effective cyber risk management policy that helps organizations protect themselves from cyber threats such as ransomware, phishing and other cyber-attacks.

Usually, OSINT analysts have a strategy and framework in place and they set their tools and techniques based on this strategy and framework. They are not collecting every piece of information available over the Internet.

OSINT collection can be categorized into two groups: passive and active collection. While active collection is using a variety of techniques to collect data, passive collection involves the use of threat intelligence platforms (TIPs).

Most common OSINT resources are as follows;

• Metadata search
• Search engines
• Social media
• Online communities
• Email addresses
• Usernames
• People search engines
• Telephone numbers
• Online maps
• Code search
• Documents
• Images
• Videos
• Domain names
• IP addresses
• Government business records
• Virtual currencies
• Advanced Linux tools
• Geospatial research
• Data breaches leaks

Technically, anyone who knows how to use these tools and techniques to access information is said to have used this process. However, because these techniques are fluid due to technological changes, a large part of them cannot be carried out by anyone with knowledge.

OSINT tools

By using OSINT tools, analysts are able to make sense of the flood of publicly available data and turn it into usable information. Tools play an important role in the search for information, but they will not be helpful for all users unless you know how important the use of these tools is.

Let’s say your job is to find out which usernames exist on various websites, such as Facebook, Twitter, LinkedIn, and other social media sites. You run multiple tools to collect goals and related information that you can correlate and use later. Conversely, this is possible by using open source tools connected to different websites (more than I can remember) and checking all usernames and presence on a website at once.

While there are a lot of tools for OSINT, I do not recommend approaching open source intelligence as just the use of tools to find anything and everything that might be interesting or useful.

Here are the most used OSINT tools;

• OSINT Framework to assess public perception and support decision-making, and to support data collection and analysis.

• CheckUserNames is an internet OSINT instrument that can assist you to discover usernames over over 170 social media platform.

• HaveIbeenPwned is a tool that can tell you if your account(s) has been compromised or not in the past.

• BeenVerified is another tool to search people’s public records on Internet.

• Censys is a search engine to collect data about any device connected to the Internet.

• BuiltWith provides detailed information about CMS used like WordPress, Joomla, Drupal, etc, as well as full-depth Javascript and CSS libraries like jQuery, bootstrap/foundation, external fonts, the webserver type (Nginx, Apache, IIS, etc), SSL provider as well as web-hosting provider used.

• Maltego offers solutions for open source intelligence and visual link analysis.

• Recon-Ng is a full-featured reconnaissance framework designed with the goal of providing a powerful environment to conduct open-source web-based reconnaissance quickly and thoroughly.

• theHarvester is a very simple to use, yet powerful and effective tool designed to be used in the early stages of a penetration test or red team engagement. Use it for open-source intelligence (OSINT) gathering to help determine a company’s external threat landscape on the internet. The tool gathers emails, names, subdomains, IPs, and URLs using multiple public data sources.

• Shodan has servers located around the world that crawl the Internet to provide the latest Internet intelligence.

• Jigsaw is an OSINT tool that let you gather information about any company employees.

• SpiderFoot is a recon tool to launch queries over 100 public data sources to gather intelligence on generic names, domain names, email addresses, and IP addresses.

• Creepy is a geolocation OSINT Tool that offers geolocation information gathering through social networking platforms.

• Nmap is one of the most popular and widely used tools by security professionals for network discovery and security auditing.

• WebShag is a multi-threaded, multi-platform web server audit tool that lets you gather commonly useful functionalities for web server auditings like website crawling, URL scanning, or file fuzzing.

• OpenVAS (Open vulnerability assessment system) is a security framework with OpenVAS Scanner, a highly efficient agent that executes all the network vulnerability tests over the target machine.

• Fierce is a semi-lightweight scanner that helps locate non-contiguous IP space and hostnames against specified domains. It’s really meant as a pre-cursor to Nmap, Unicornscan, Nessus, Nikto, etc since all of those require that you already know what IP space you are looking for.

• Unicornscan is an OSINT tool that can scan TCP/IP device/network, UDP Protocol, identify A/P OS, detect application and component and support SQL Relational Output.

• FOCA (Fingerprinting Organizations with Collected Archives) is a tool used mainly to find metadata and hidden information in the documents its scans. These documents may be on web pages and can be downloaded and analyzed with FOCA.

• ZoomEye is an IoT OSINT search engine that uses WMAP and XMap in order to build its database.

• Spyse holds the largest database of its kind, containing a wide range of OSINT data handy for the reconnaissance.

• IVRE (French: Instrument de veille sur les réseaux extérieurs) or DRUNK (Dynamic Recon of UNKnown networks) is an open-source framework for network recon, written in Python. It relies on powerful open-source tools to gather intelligence from the network, actively or passively.

• Metagoofil is an information gathering tool designed for extracting metadata of public documents (pdf,doc,xls,ppt,docx,pptx,xlsx) belonging to a target company.

• Exiftool is a platform-independent Perl library plus a command-line application for reading, writing, and editing meta information in a wide variety of files. ExifTool supports many different metadata formats including EXIF, GPS, IPTC, XMP, JFIF, GeoTIFF, ICC Profile, Photoshop IRB, FlashPix, AFCP, and ID3, as well as the maker notes of many digital cameras by Canon, Casio, DJI, FLIR, FujiFilm, GE, GoPro, HP, JVC/Victor, Kodak, Leaf, Minolta/Konica-Minolta, Motorola, Nikon, Nintendo, Olympus/Epson, Panasonic/Leica, Pentax/Asahi, Phase One, Reconyx, Ricoh, Samsung, Sanyo, Sigma/Foveon and Sony.

Other than these tools, with the help of artificial intelligence, OSINT analysts are able to meaningfully capture streams of publicly available data and transform them into usable open-source intelligence.

To conclude, OSINT tools play an important role in the search for information, and they will be helpful not only for the user to know the importance of using these tools but also to understand what is available on the market. Keep an eye on open source information – a compelling perspective from the perspective of an analyst or researcher.