OSINT Technologies: Safe Online Data Collection
Open source intelligence (OSINT) is considered publicly available information (PAI). OSINT, until the internet was introduced, included newspaper clippings and library resources. OSINT mediums are ever-changing, and therefore, collecting this information is both an art and a science. Finding the most effective and secure OSINT technologies have a lasting impact on investigations.
Conveying the sentiment coined by William Gibson, “the street finds its own uses for things,” OSINT technologies must change even faster than the mediums that contain it. Investigators require technology with a predictive development cycle, one that allows them to outrun adversaries to their next playing field.
OSINT Technologies for a Multi-Layered Environment
The internet, what we now find to be the backbone and start of all OSINT investigations, is multi-layered.
Surface Web
The first layer is the surface web, where we all spend our daily lives. The surface web doesn’t look the same from every angle, meaning depending on who website administrators think you are, where you’re located, and what your browsing history looks like, you will often be fed different information.
There are endless surface web tools that provide access to various categories of data. For example, Google Maps provides geographic data, Translate This provides linguistic data, and archiving tools provide historical data. Like this, each search engine retrieves different search results. Meta search engines even query and aggregate data across search engines.
Many of these tools come in the form of browser add-ons. It’s important to be aware that these add-ons slow browsers down and are often outdated and prone to breaking, creating a doorway of vulnerabilities to your investigation. OSINT technologies with integrated surface web tools close these vulnerable gaps.
Deep Web
The second layer of the internet is the deep web. Visually, the topography of the deep web is the most complex. But data in the deep web is not abstract; it is simply not indexed by search engines and just requires authentication to reach it.
Each deep web platform caters to a different user group. Think of deep web platforms as unique networks of people. Since OSINT is most often for the purpose of researching an individual, shifting the investigative mindset to researching networks rather than individuals will seem less like finding a needle in a haystack.
Additionally, effective online criminals are members of various platforms, which muddies the network waters. Investigators must move laterally through networks, analyzing the degree of separation and looking for weak links in order to gain access to the core individual. Networks, although not directly connected across platforms, are interconnected on a social level. And each platform serves a separate purpose. Without an understanding of social sciences, navigating this terrain can be hazardous during an investigation. OSINT technologies that are developed specifically for securely navigating deep web networks facilitates safe investigations.
The Dark Web
The third layer of the internet, which is known often as the most dangerous, is the dark web, accessed through the Tor browser. Growing interest by investigators in the dark web has required a new level of training to fully understand the behavior of Tor and its intricacies. Investigators must be able to connect information on the dark web to real-world people, places, and things.
Navigating Tor as an investigator without employing a managed attribution (MA) platform is like walking down the street in plain clothes with your badge in your back pocket. Pickpockets know you’re there, and if you aren’t paying attention, will reveal your identity and track your footprints. Investigators must follow a “never trust” philosophy while navigating and collecting OSINT on the dark web and employ holistic OSINT technologies as a foundational layer of security.
Cyber Geography Matters
Just like world geography, the internet is broken down into cyber geography. The domain name system (DNS) both defines the geographic breakdown and makes URLs readable by humans. Top-level domains (TLDs)—like org, com, and gov—further categorize what may be found at the URL address. Without even a high-level understanding of URLs, huge volumes of data can be missed by an investigator researching in the wrong cyber region.
Further, the search results retrieved by search engines very much depend on the user’s location and search history. There are ways around coaxing neglect of geographies, such as using Google NCR or using a clustering search engine like Carrot2. Search results can also be refined in other ways using search operators, known as Google Dorking.
Key OSINT Technologies
Operational Security Rigor
The internet was built for efficiency, not security. There are two types of security that investigators must adopt: practical and technical. Although OSINT technologies are advancing overnight to add layers of security to investigations, humans will always be the weakest link. As layers are added, developers must be cognizant of the diversity, flexibility, and usability of technology. Striking that balance is the difference between simply securing an operation and advancing the performance and efficiency of an operation without sacrificing security.
Here are some ideas to keep in mind when considering the use of OSINT technologies. We’ll expand on these ideas more throughout this year.
- An air-gapped managed attribution platform inside a VDI provides a critical separation of the OSINT investigation’s activity from your organization’s network.
- Never cross-contaminate research machines with normal browsing activity.
- Good computer hygiene should be a regular practice.
- As soon as you use social media, the platform has full permission to track your activity. Always log out of social media platforms when day-to-day research is complete.
With these key foundational components, investigators can avoid the direct observation of their online activities and affiliated organizations. But an investigator can’t be totally invisible. Browser profiles and IP addresses are unique and visible to web servers and website administrators as digital fingerprints. With virtual machines (VM), malware and trackers are destroyed at the end of every session through the purging of the VM. MA-platform use alongside safe practices protect the investigator, organization, and research.
Going Dark with Smart Security
Complete privacy on the internet is an oxymoron. The internet’s sole purpose is communication. IP addresses allow systems to communicate to one another in order to retrieve the data that you need and navigate to where you want to go. It’s as easy to figure out your own IP address as it is to figure out another’s.
Why not just use Tor with all its embedded nodes to effectively hide online? Because OSINT investigators need to do more than just blend in. Using Tor is a sure sign to website administrators that you are trying to hide from someone. This may lead to purposeful misinformation by website administrators. Additionally, because of its complex network of nodes, Tor is slow.
Enter VPNs: combining the use of VPNs with an MA platform, investigators can access geographically specific information from the IP location of choice, whereas this can’t be done using Tor. Investigators can even build a custom network rather than rely on the unreliable alternatives. OSINT technology stretches far beyond the walls of your facility and the computer on your desk.
Smart security involves using the right OSINT technology and techniques at the right time—without inhibiting an operation’s diversity, flexibility, and utility. Passive casual browsing requires an MA platform and VPNs. More advanced online operations require access to integrated third-party tools and custom applications within the MA platform. Furthermore, high-pressure situations require intensive practical training and preparation.
Conclusion
Navigating, analyzing, and collecting OSINT on the multi-layered internet need a multi-layered security approach. With these OSINT technologies and techniques and those that will follow in 2022’s Ntrepid blog series, The Art of Operating Online, investigators will learn the critical methods for safely advancing ahead of adversaries.
