Grey bar Blue bar
Share this:

Fri, 13 Jun 2014

Using Maltego to explore threat & vulnerability data

This blog post is about the process we went through trying to better interpret the masses of scan results that automated vulnerability scanners and centralised logging systems produce. A good example of the value in getting actionable items out of this data is the recent Target compromise. Their scanning solutions detected the threat that lead to their compromise, but no humans intervened. It's suspected that too many security alerts were being generated on a regular basis to act upon.

The goal of our experiment was to steer away from the usual data interrogation questions of "What are the top N vulnerabilities my scanner has flagged with a high threat?" towards questions like "For how many of my vulnerabilities do public exploits exist?". Near the end of this exercise we stumbled across this BSides talk "Stop Fixing All The Things". Theses researchers took a similar view-point: "As security practitioners, we care about which vulnerabilities matter". Their blog post and video are definitely worth having a look at.

At SensePost we have a Managed Vulnerability Scanning service (MVS). It incorporates numerous scanning agents (e.g. Nessus, Nmap, Netsparker and a few others), and exposes an API to interact with the results. This was our starting point to explore threat related data. We could then couple this data with remote data sources (e.g. CVE data, data).

We chose to use Maltego to explore the data as it's an incredibly powerful data exploration and visualisation tool, and writing transforms is straight forward. If you'd like to know more about Maltego here are some useful references:

What we ended up building were:

  • Transforms to explore our MVS data

  • A CVE / API engine

  • Transforms to correlate between scanner data and the created APIs

  • Maltego Machines to combine our transforms

So far our API is able to query a database populated from CVE XML files and data from (they were kind enough to give us access to their CVE inclusive data set). It's a standalone Python program that pulls down the XML files, populates a local database, and then exposes a REST API. We're working on incorporating other sources - threat feeds, other logging/scanning systems. Let us know if you have any ideas. Here's the API in action:

Parsing CVE XML data and exposing REST API
Parsing CVE XML data and exposing REST API

Querying a CVE. We see 4 public exploits are available.
Querying a CVE. We see 4 public exploits are available.

It's also worth noting that for the demonstrations that follow we've obscured our clients' names by applying a salted 'human readable hash' to their names. A side effect is that you'll notice some rather humorous entries in the images and videos that follow.

Jumping into the interesting results, these are some of the tasks that we can perform:

  • Show me all hosts that have a critical vulnerability within the last 30 days

  • Show me vulnerable hosts for which public exploit code exists

  • Show me all hosts for which a vulnerability exists that has the word 'jmx-console' in the description

  • Show me all hosts on in my DMZ that have port 443 open

  • Given a discovered vulnerability on a host, show me all other hosts with the same vulnerability

  • Show me a single diagram depicting every MVS client, weighted by the threat of all scans within the last week

  • Show me a single diagram depicting every MVS client, weighted by the availability of public exploit code

  • Given a CPE, show me all hosts that match it

Clicking the links in the above scenarios will display a screenshot of a solution. Additionally, two video demonstrations with dialog are below.

Retrieving all recent vulnerabilities for a client 'Bravo Tango', and checking one of them to see if there's public exploit code available.
Retrieving all recent vulnerabilities for a client 'Bravo Tango', and checking one of them to see if there's public exploit code available.

Exploring which clients/hosts have which ports open
Exploring which clients/hosts have which ports open

In summary, building 'clever tools' that allow you to combine human insight can be powerful. An experiences analyst with the ability to ask the right questions, and building tools that allows answers to be easily extracted, yields actionable tasks in less time. We're going to start using this approach internally to find new ways to explore the vulnerability data sets of our scanning clients and see how it goes.

In the future, we're working on incorporating other data sources (e.g. LogRhythm, Skybox). We're also upgrading our MVS API - you'll notice a lot of the Maltego queries are cumbersome and slow due to its current linear exploration approach.

The source code for the API, the somewhat PoC Maltego transforms, and the MVS (BroadView) API can be downloaded from our GitHub page, and the MVS API from here. You'll need a paid subscription to incorporate the data, but it's an initiative definitely worth supporting with a very fair pricing model. They do put significant effort in correlating CVEs. See this page for more information.

Do get in touch with us (or comment below) if you'd like to know more about the technical details, chat about the API (or expand on it), if this is a solution you'd like to deploy, or if you'd just like to say "Hi".

Mon, 4 Mar 2013

Vulnerability Management Analyst Position

Have a keen interest on scanning over 12000 IP's a week for vulnerabilities? Excited about the thought of assessing over 100 web applications for common vulnerabilities? If so, an exciting, as well as demanding, position has become available within the Managed Vulnerability Scanning (MVS) team at SensePost.

Job Title: Vulnerability Management Analyst

Salary Range: Industry standard, commensurate with experience

Location: Johannesburg/Pretoria, South Africa

We are looking for a talented person to join our MVS team to help manage the technology that makes up our Broadview suite and, more importantly, finding vulnerabilities, interpreting the results and manually verifying them. We are after talented people with a broad skill set to join our growing team of consultants. Our BroadView suite of products consists of our extensive vulnerability scanning engine, which looks at both the network-layer and the application layer, as well as our extensive DNS footprinting technologies.

The role of the Vulnerability Management Analyst will possess the following skills:

  • Be able to multitask and meet client deadlines. We want a person that thinks 'I can do that!'

  • Possess excellent written and oral communication skills. Being able to understand a vulnerability and explain it to business leaders is a must.

  • A working knowledge of enterprise vulnerability management products and remedial work flow

  • A broad knowledge of most common enterprise technologies and operating systems

  • A passion for security and technology

Some additional conditions:

  • A post graduate degree or infosec certification would be beneficial, however, showing us you have the passion and skills also helps

  • This job requires some after-hours and weekend commitments (we try to keep this to a minimum)

  • Bonus points for knowledge of sed, awk and python, ok even ruby.

  • PCI-QSA is desired but not required

Impress us with your skills by sending an email to and lets take it from there.

SensePost is an equal opportunity partner.

Wed, 9 Mar 2011

You got to want it bad

In the movie "The American President", the statement is made that America has advanced citizenship and that "you gotta want it bad, because it will put up a fight". The same can be said for vulnerability management. It is never a completed exercise or a process where the status quo can be maintained quite easily, especially in a distributed enterprise environment. The reason: change.

SensePost recognised early on that just having an accurate vulnerability scanner isn't good enough to ensure continuous and less arduous vulnerability management. There needs to be workflow and efficiency build into such a scanner. Hence our HackRack and now lately, our BroadView managed vulnerability scanning offerings.

But, no matter how good a scanner is or how well the workflow has been designed, there is still a very large amount of manual analysis required.

For example:

In BroadView, when viewing scan results, by default the Medium, High and Critical findings are shown. Fab and groovy. But, should one just stop there? The Low and Info findings can be as interesting as the rest. For example, a client of ours that usually has a handle on things, had an informational finding about virtual directories being guessable on one of their web servers: the directories "/testing" and "/test" were identified. This "/testing" directory turned out to contain the beta version of a new e-commerce web application and even though reasonable security was in place, a blind SQL injection test showed us they were developing on live data. Just like that, an informational finding became a critical finding. If we had been focused on CVSS scores and risk impact only, this finding would have been flying under the radar.

What we saw on BroadView:

Vulnerability management is not easy. It will put up a fight; be that in the form of stubborn sysadmins not closing the holes or developers taking chances with release candidates and beta products. The vulnerability manager has to be on his/her toes and perform constant scanning and prodding. Vulnerability scanner results should never be taken at face value, and the associations between findings should be understood.

It is wise to keep in mind that vulnerability management is cyclic and repetitive. And as Dr Ruth always used to say: "Once, is not enough". You cannot scan once, find nothing, and sit back and relax. You may just miss your /testing directory.

For our BroadView customers we have added a couple of new blizzards to enhance the process to monitor results.

  • Missing Microsoft Patches (Operating System category)
  • Guessable Virtual Directories (Web Application category)
  • Open jBoss Consoles (Web Application category)
Blizzards are widgets (iGoogle style) of information queried from the vulnerability database in BroadView that provides users with a looking glass view of their environment. Under normal circumstances one would have had to go grep or search for very specific vulnerability IDs. With the blizzards, that cumbersome task has been removed.

The Missing Microsoft Patches blizzard combines all the possible patches that could be missing and this is especially necessary where Internet facing targets are scanned. Murphy's Law usually applies where patches and Internet facing devices are concerned - that one patch that can result in pwnage, is normally the one missing.

The output from the Missing Microsoft Patches blizzard would typically consist of an IP:Value output

The jBoss Console blizzard was created after we realised it is becoming more and more prevalent for consoles to be found open during assessments and vulnerability scanning.

Having access to world class pen-testers really does give the vulnerability management team a good insight into which vulnerabilities can actually lead to system compromise.

Happy scanning

Wed, 2 Mar 2011

To understand the battlefield, you need a broad view

It is always a little bemusing to hear that we only provide pentests. Since 2001, SensePost has offered a very comprehensible vulnerability management service that's evolved through multiple generations of technologies and methodologies into a service we're very proud of. The Managed Vulnerability Scanning ("MVS") service makes use of our purpose-built BroadView scanning technology to scan a number of high profile South African and European clients. More information can be found here, but the purpose of this post is to introduce it with a basic overview of its deployment.

To give you a better understanding of our coverage, below are a number of statistics from our scanning database.

Number of scans per week: 935 average per week

Number of findings stored: 3 795 963

Number of collected attribute instance: 1 274 016

Number of unique IPs listed as targets: 24723

Number of unique IPs with issues: 4931

However, the stats are not the interesting bit. BroadView goes further than simply storing open issues, it also tags interesting characteristics of the targets using 'attributes', which are pieces of information associated with a finding, but are not necessarily a result. It is possible to query these attributes and tie them back to hosts; this enables you to search across all hosts for matching attributes. The most used attributes are:

  • TCP Banners
  • Operating System Value
  • Hosts Accessible (True/False)
  • SMTP Relaying Allowed (True/False)
  • SMB Directories
  • CMS Type
With all these attributes, one can perform intelligent scanning or reporting. For example, target all Windows devices with an open port 80 and running IIS5, or show a list of all open relays on our domain, or keep an updated list that shows all BIND servers that still require the recent DoS patch. This can be very useful, especially when setting up targeted scans or for network/patch management. Effectively, the attributes allow you to utilize BroadView as a network service monitoring device rather than just a vulnerability scanner. BroadView makes use of a dashboard to display blizzards (widgets with specific data sets); the data source for the blizzards is anything we can pull from the vulnerability and attribute database, displayed as a list or graph. For this purpose we have specific widgets that can show you in an instant the open ports across your network, sensitive open ports such as database services or phpmywebadmin instances.

So, we have loads of data and it makes for interesting analysis.

For example:

The number of targets with potential webservers: 918

And breaking it down further:

  • Apache =186
  • IIS = 303
The number of targets inviting worm trouble: (port 139 open to the Internet)

The top 3 SSL certificate issuers used:

  • Entrust - 230
  • VeriSign - 159
  • Thawte - 47
And many more.

Next time, more about the dashboard and the blizzards.

Tue, 6 Apr 2010

BroadView V4 Attributes

Following on from Evert's posting about the new BroadView v4, I'd like to showcase a specific aspect of BV that we've found useful, namely Attributes. These are small pieces of data collected and maintained for each host scanned by BV including somewhat mundane bits of info like IP address and OS but, they also include some really tasty morsels about remote hosts that are scanned. Attributes are collected on a per-scan-per-host basis, and are populated by each test that runs during the scan. Since attribute population is dependent on the selected tests, the set of Attributes available to you would vary according to you configuration.

Consider the trivial attribute Network.TCP.HTTP.Banner; this doesn't require credentials to acquire and is stored by a test that detects webservers. On the other hand, the test that stores Users.Microsoft.Windows.Group.SystemOperators.Members would require domain credentials in order to pull the needed info. This is common inside of organisations, where BV is primarily intended.

To help me explain the power of Attributes a little easier, here are a few scenarios:

Your IT manager wants to know which Windows machines are missing the new MS10-018 patch. Instead of trawling through all the latest scans looking for hosts that are affected , you simply:

  1. Login to BroadView
  2. Click Attributes
  3. Select Patches.Microsoft.Windows.Missing
  4. Click MS10-018
  5. Download CSV
  6. Done
Perhaps you have rolled-out a new WSUS system and need to find all the Windows hosts still configured with the old WSUS server name. Again:
  1. Login to BroadView
  2. Attributes
  3. Config.Microsoft.Windows.WSUS.Server
  4. Click the name of the old WSUS server
  5. Download CSV
  6. Done
Or you are trying to find all the hosts with a specific piece of software installed (e.g. uTorrent). Click Attributes >> Software.Installed.Microsoft.Windows >> uTorrent >> Download CSV.

One of the IT techies gives you a call:

Bob: Hey Steve Steve: Ahoy Bob: Do you know which FTP servers on the network allow Anonymous access? Steve: Ofcourse I do Login to BroadView >> Attributes >> Network.TCP.FTP.IsAnonymousAccessAllowed >> True >> Download CSV Steve: You got mail Bob: Awesome, thanks

As you can see the power and extensibility of BroadView Attributes is (according to opinions from the office) Simply Astonishing(tm). We are currently working with our Assessment team to include Attributes that would allow them to very quickly pull a list of all "low hanging fruit" vulnerabilities when performing an internal Pen Test.

Currently we collect just over 50 attributes, but are adding new ones as we either think of or clients request more. The full list is:

Services.Microsoft.Windows.Running Users.Microsoft.Windows.Local.LastLoggedIn Users.Microsoft.Windows.Local.NeverLoggedIn Users.Microsoft.Windows.Local.PasswordNeverExpires Users.Microsoft.Windows.Group.AccountOperators.Members Users.Microsoft.Windows.Group.BackupOperators.Members Users.Microsoft.Windows.Group.PrintOperators.Members Users.Microsoft.Windows.Group.Replicators.Members Users.Microsoft.Windows.Group.SystemOperators.Members Users.Microsoft.Windows.Network.NeverChangedPasswords Users.Microsoft.Windows.Network.NeverLoggedOn Users.Microsoft.Windows.Network.PasswordNeverExpires Users.Microsoft.Windows.ActiveDirectory.Group.Members Users.Microsoft.Windows.ActiveDirectory.AccountsOld.Members Users.Microsoft.Windows.ActiveDirectory.AccountsStale.Members Users.Microsoft.Windows.ActiveDirectory.AccountsBadLogins.Members Users.Microsoft.Windows.ActiveDirectory.AccountsOldPassword.Members Users.Microsoft.Windows.ActiveDirectory.AccountsPasswordNeverSet.Members Users.Microsoft.Windows.ActiveDirectory.AccountsDisabled.Members Users.Microsoft.Windows.ActiveDirectory.AccountsLocked.Members Config.Microsoft.Windows.Domain.IsCorrect Config.Microsoft.Windows.Domain.Value Config.Microsoft.Windows.WSUS.Server Config.Microsoft.Windows.WSUS.Server.IsConfigured Config.Microsoft.Windows.WSUS.Server.Value Config.Microsoft.Windows.MachineName Debug.Network.IsHostAccessible
Debug.Microsoft.Windows.Registry.Access.Full Debug.Microsoft.Windows.Registry.Access.Read Debug.Microsoft.Windows.Registry.Access.Fail Debug.Microsoft.Windows.Privileges.Admin.Full Debug.Microsoft.Windows.Privileges.Admin.Fail ServicePacks.Microsoft.Windows.Win2k3.Value ServicePacks.Microsoft.Windows.Win2k3.IsInstalled ServicePacks.Microsoft.Windows.NT4.Value ServicePacks.Microsoft.Windows.NT4.IsInstalled ServicePacks.Microsoft.Windows.Win2k.Value ServicePacks.Microsoft.Windows.Win2k.IsInstalled ServicePacks.Microsoft.Windows.XP.Value ServicePacks.Microsoft.Windows.XP.IsInstalled Software.Microsoft.Office.Value Software.Microsoft.Office.IsInstalled Software.Microsoft.SMSAgent.IsInstalled Software.Microsoft.SMSAgent.IsRunning Software.Microsoft.SMSAgent.IsInstalled Software.Microsoft.SMSAgent.McAfee.EPOAgent.IsInstalled Software.AntiVirus.Linux Processes.Microsoft.Windows Network.TCP Network.TCP.FTP.IsAnonymousAccessAllowed Network.TCP.SMTP.IsRelayAllowed Network.TCP.HTTP.Banner Network.TCP.HTTP.Directories Network.TCP.Banner Network.TCP.SMB.Direcotories Network.UDP.DNS.ReverseDNS Network.UDP.LDAP.BaseObject