Grey bar Blue bar
Share this:

Sat, 7 Aug 2010

BlackHat Write-up: go-derper and mining memcaches

[Update: Disclosure and other points discussed in a little more detail here.]

Why memcached?

At BlackHat USA last year we spoke about attacking cloud systems, while the thinking was broadly applicable, we focused on specific providers (overview). This year, we continued in the same vein except we focused on a particular piece of software used in numerous large-scale application including many cloud services. In the realm of "software that enables cloud services", there appears to be a handful of "go to" applications that are consistently re-used, and it's curious that a security practitioner's perspective has not as yet been applied to them (disclaimer: I'm not aware of parallel work).

We choose to look at memcached, a "Free & open source, high-performance, distributed memory object caching system" 1. It's not outwardly sexy from a security standpoint and it doesn't have a large and exposed codebase (total LOC is a smidge over 11k). However, what's of interest is the type of applications in which memcached is deployed. Memcached is most often used in web application to speed up page loads. Sites are almost2 always dynamic and either have many clients (i.e. require horizontal scaling) or process piles of data (look to reduce processing time), or oftentimes both. This implies that the sites that use memcached contain more interesting info than simple static sites, and are an indicator of a potentially interesting site. Prominent users of memcached include LiveJournal (memcached was originally written by Brad Fitzpatrick for LJ), Wikipedia, Flickr, YouTube and Twitter.

I won't go into how memcached works, suffice it to say that since data tends to be read more often than written in common use cases the idea is to pre-render and store the finalised content inside the in-memory cache. When future requests ask for the page or data, it doesn't need to be regenerated but can be simply regurgitated from the cache. Their Wiki contains more background.


We released go-derper, a tool for playing with memcached instances. It supports three basic modes of operations:
  1. Fingerprinting memcacheds to determine interesting servers
  2. Extracting a (user-limited) copy of the cache
  3. Writing data into the cache
The tool has minor requirements: a recent Ruby and the memcache-client gem. What follows are basic use cases.


Let's assume you've scanned a hosting provider and found 239 potential targets using a basic .nse that hunts down open memcached instances3. You need to separate the wheat from the chaff and figure out which servers are potentially interesting; one way to do that is by extracting a bunch of metrics from each cache. Start small against one cache: insurrection:demo marco$ ruby go-derper.rb -f x.x.x.x [i] Scanning x.x.x.x x.x.x.x:11211 ============================== memcached 1.4.5 (1064) up 54:10:01:27, sys time Wed Aug 04 10:34:36 +0200 2010, utime=369388.17, stime=520925.98 Mem: Max 1024.00 MB, max item size = 1024.00 KB Network: curr conn 18, bytes read 44.69 TB, bytes written 695.93 GB Cache: get 514, set 93.41b, bytes stored 825.73 MB, curr item count 1.54m, total items 1.54m, total slabs 3 Stats capabilities: (stat) slabs settings items (set) (get)

44 terabytes read from the cache in 54 days with 1.5 million items stored? This cache is used quite frequently. There's an anomaly here in that the cache reports only 514 reads with 93 billion writes; however it's still worth exploring if only for the size.

We can run the same fingerprint scan against multiple hosts using

ruby go-derper.rb -f host1,host2,host3,...,hostn

or, if the hosts are in a file (one per line):

ruby go-derper.rb -F file_with_target_hosts

Output is either human-readable multiline (the default), or CSV. The latter helps for quickly rearranging and sorting the output to determine potential targets, and is enabled with the "-c" switch:

ruby go-derper.rb -c csv -f host1,host2,host3,...,hostn

Lastly, the monitor mode (-m) will loop forever while retrieving certain statistics and keep track of differences between iterations, in order to determine whether the cache appears to be in active use.


Once you've identified a potentially interesting target, it's time to mine that cache. The basic leach switch is "-l":

insurrection:demo marco$ ruby go-derper.rb -l -s x.x.x.x [w] No output directory specified, defaulting to ./output [w] No prefix supplied, using "run1"

This will extract data from the cache in the form of a key and its value, and save the value in a file under the "./output" directory by default (if this directory doesn't exist then the tool will exit so make sure it's present.) This means a separate file is created for every retrieved value. Output directories and file prefixes are adjustable with "-o" and "-r" respectively, however it's usually safe to leave these alone.

By default, go-derper fetches 10 keys per slab (see the memcached docs for a discussion on slabs; basically similar-sized entries are grouped together.) This default is intentionally low; on an actual assessment this could run into six figures. Use the "-K" switch to adjust:

ruby go-derper.rb -l -K 100 -s x.x.x.x

As mentioned, retrieved data is stored in the "./ouput" directory (or elsewhere if "-o" is used). Within this directory, each new run of the tool produces a set of files prefixed with "runN" in order to keep multiple runs separate. The files produced are:

  • runN-index, an index file containing metadata about each entry retrieved
  • runN-<md5>, a file containing the bytestream from a retrieved value
The mapping between key and file in which the value is stored occurs in the index file, which is useful in that potentially malicious data (keynames) aren't used when interacting with your local filesystem APIs.

At this point, there will (hopefully) be a large number of files in your output directory, which may contain useful info. Start grepping.

What we found with a bit of field experience was that mining large caches can take some time, and repeating grep gets quite boring. The tool permits you to supply your own set of regular expressions which will be applied to each retrieved value; matches are printed to the screen and this provides a scroll-by view of bits of data that may pique your interest (things like URLs, email addresses, session IDs, strings starting with "user", "pass" or "auth", cookies, IP addresses etc). The "-R" switch enables this feature and takes a file containing regexes as its sole argument:

ruby go-derper.rb -l -K 100 -R regexs.txt -s x.x.x.x


In this blog entry I don't cover the kinds of data we discovered (it'll be subject to a separate entry), however it may come to pass that you discover an interesting cache entry that you'd like to overwrite. Recall entries were stored in "./output" by default, with a prefix of "runN". If the interesting entry was stored in "output/run1-e94aae85bd3469d929727bee5009dddd", edit the file in whatever manner you see fit and save it to your local disk. Then, tell go-derper to write the entry back into the cache with:

ruby go-derper.rb -w output/run1-e94aae85bd3469d929727bee5009dddd

This syntax is simple since go-derper will figure out the target server and key from the run's index file.

And so?

Go-derper permits basic manipulations of a memcached instance. We haven't covered finding open instances or the kinds of data one may come across; these will be the subject of followup posts. Below are the slides from the talk, click through to SlideShare for the downloadable PDF.

2 We're hedging here, but we've not come across a static memcached site.

3 If so, you may be as surprised as we were in finding this many open instances.

Thu, 11 Jun 2009

Apple vs Microsoft as a malware target.. stop saying market share..

I really enjoy listening to Mac Break Weekly.. Leo Laporte is an excellent host and i would tune in just to hear [Andy Ihnatko's] take on the industry and the (possible) motivations behind certain players moves. (he is sometimes wrong, but always worth listening to). The only time the things ever get a little cringe-worthy is when talk switches to malware and security (although both Andy and Leo for the most part have pretty reasonable balanced views on it).

Disclosure: I am a mac user, and love the hardware.. the fan-boy'ism that surrounds it, not so much..

Most security savvy mac users, dont push Invulnerable-Mac argument too much.. But it does lead to the follow-up "Once Mac gets more market share, we will hit the malware tipping point".. I dont think that this is how it will go down.. Here's my $0.002c on it.

One of the talks we gave at the recent ITWeb Security Summit was titled "One bad Apple".. The aim of the talk was to examine the truth/lies/fud behind the security claims on both the fan-boy and hater end of the spectrum.. I dont want to cover the whole talk here, but do want to touch on just a few of the current annoying red-herrings that normally pop up in this discussion:

Vulnerability counts as a useful Metric

This argument has been had by [many people] far brighter than me, so i wont rehash it here. I think its safe to say that since there isnt really a standard on what gets reported, very few vuln count reports end up comparing apples with apples. What i did pick on during the talk, was that some people dont even bother trying to dress up the stats in a cloak of reasonableness. The table below was taken from ByteSize magazine showing that Apple indeed had more Vulnerability Disclosures than Microsoft:

Vendors with the Most Vulnerability Disclosures (ByteSize - 3rd Ed. 2009)

Instead of muddying the water by asking what a 3.2% disclosure means, or by comparing Apple with Microsoft you have to ask yourself if the table is really comparing Microsoft, with its software, hardware, * against Wordpress with its 60 000 lines of PHP code?

My suggestion there is that if we going to use tables and charts, we should at least stick to the reasonable ones:

Malware defense

Of course the next topic that refuses to die is how mac architecture pixie-dust prevents it from getting worms and viruses.. A quick check should clarify this.. The ILOVEYOU virus which took windows computers all over the world (and according to Wikipedia cost about $5.5 billion in damage) was a snippet of VBS that read your address book, and mailed itself to your contacts (where it did the same). You can hack this up in Automator in seconds.. Same functionality completely..

Memory Corruption Attacks

In recent times, Microsoft has made huge leaps in terms of generic memory corruption protection mechanisms to minimize the effect of buffer overflow/mem corruption attacks. While Apple claimed to do the same with Leopard, they still trail Microsoft in this regard. The 3 points we covered:

  1. Non-executable Stack.
  2. Non-executable Heap.
  3. Address Space Layout Randomization.

(We cover these in more detail in an upcoming [conference in July] - but again, its fairly well understood that OSX in its current form is only randomizing libraries, and that to get the benefit of ASLR, you need to be randomizing everything)

So if we are saying that Apple is just as vulnerable to ILOVEYOU and even more vulnerable today than Windows from a nimda or a code-red, then what explains the fact that we dont see Macs getting owned on the same level as Windows?

The almost global answer is "Market share!". The belief that once more people are running macs, the big bad malware writers will start aiming at them.

If you look at the [netcraft web server survey] (2003) you should notice that at the time that nimda and code-red were running around the Internet, IIS didnt have the lions share of the webserver market either. Their lower market share didnt keep them safe then, why does it keep mac users safer now ?

The real market share difference

One of my guesses here is that we are looking at the wrong data for market share. What Microsoft does have over Apple, is a bigger market share of [developers..]

Microsoft went out of their way to make sure that anyone and their dog could write code for their platform, that any idiot in the world could write an app for them, and many did. I suspect that if you consider that any group will have a proportion of people with evil intentions, then in part what we seeing is just the percentage of the bigger pool.

Different user profiles

The other thing (although it sounds strange) is the question of user culture which is different. My wifes macbook air has very little software that didnt come with the machine. Apples "batteries included" policy means that her machine remains pretty clean.. Her mothers windows machine is a different story

Which means what?

Today, pound for pound, OS X Leopard is indeed more vulnerable than a Vista machine, but the eco system around Mac is holding back the huge embarrassing attacks that shamed Microsoft into action. Apple has a small window during which time they can take action, refine their built in mitigation strategies and come out on the other side acting like they were better all along..

(Recent hires like Ivan give hope for this happening)

If Snow Leopard is done right, it will hopefully be Apples XP-SP2, and us fanboys will be able to keep our securer-than-thou attitude.. If it doesnt, its only a matter of time..

Wed, 13 May 2009

Apple gets some clue points?

At [DeepSec] last year i had the pleasure of hearing Ivan Krsti? speak. While some of his arguments had (small) holes in them (which the audience were quick to pounce on), he raised the ugly fact that people like me like to ignore.. That some of us spend a lot more time thinking of elaborate ways to break stuff than we do designing less breakable stuff..

I think for most security "breakers" its an argument that sometimes hits hard, and makes you wonder if you should be refocusing your efforts..

Ivan designed the bitfrost security system for the OLPC and is/was a Harvard academic with strong ties to the Python community. (you can follow his talk schedule here).

It seems, he has just taken a position at [Apple]

We recently wrote a paper contrasting the built in memory protection mechanisms on OSX and its windows counterparts, and concluded the paper with the following lines:

"It can be postulated that OS X currently sits in an unusual niche, staying off the radar of server-attackers while below the threshold to make it an attractive target for attackers wishing to capture large volumes of desktop computers (for botnets or similar activities). Apple would be well advised to make good use of their time in this niche to learn from the mistakes made by those before them, because as their market share steadily rises, they steadily inch closer to moving out of this protected space.... .. We hope that Apple is able to make the necessary improvements before it too is forced into altering its views on generic OS protection mechanisms through the media frenzy that follows public security breaches."

It would seem like with a move like this, Apple are thinking these thoughts too..


Fri, 23 Jan 2009

QoW: Software Reversing and Exploitation

I've developed a FTP like multi-threaded server application as a target for this challenge of the month. It has been coded in c and compiled by VC++ 2008. This is a three step challenge:

Step 1- Find the correct "passphrase" format to logon to the server and get the "Access Granted" message. (You may use a debugger like Ollydbg to do Live RE for this step).

Step 2- Do vulnerability research on the server software. There is at least one exploitable bug but there could be more bugs or error conditions. Try to spot a memory corruption bug and write a denial of service exploit for it.

Step3- Convert your DoS exploit to a code execution exploit to get a connect-back shell.

If you have questions on the challenge, post them here (or to behrang AT

[you should be able to run the server on just about anything - bug will be exploitable even under XP-SP*]


Mon, 29 Dec 2008

Dont look now, but it seems they broke the Interwebs again..

Those pesky hackers!

Alex Sotirov (of heap feng shui fame, famous for breaking everything from Vista, to web browsers, to facebook) and Jacob Applebaum (of cold-boot attack fame, and more importantly of "knuth is my homeboy" fame) will be talking in a few hours at the 25c3 conference in Germany and by all accounts its going to be an "Internet Breaker".

There is a fair bit of speculation on the nature of the bug (though most people some confident that its routing protocol related) and HD Moore has blogged that the pair have sought legal advice pre-publishing.

If i had to, i would take a guess at BGP too, mainly because the talk is labeled "Making the theoretical possible" which was a tagline used by the l0pht back when they were talking about shutting down the internet with BGP related attacks.

The only problem i have with all this, is that it reveals confusion over how we measure "the year" when we award pwnies.. if the talk happens on the last day (just about) of 2008.. Does it count for pwnies 09??