[Update: Disclosure and other points discussed in a little more detail here.]

Why memcached?

At BlackHat USA last year we spoke about attacking cloud systems, while the thinking was broadly applicable, we focused on specific providers (overview). This year, we continued in the same vein except we focused on a particular piece of software used in numerous large-scale application including many cloud services. In the realm of "software that enables cloud services", there appears to be a handful of "go to" applications that are consistently re-used, and it's curious that a security practitioner's perspective has not as yet been applied to them (disclaimer: I'm not aware of parallel work).

We choose to look at memcached, a "Free & open source, high-performance, distributed memory object caching system" ¹. It's not outwardly sexy from a security standpoint and it doesn't have a large and exposed codebase (total LOC is a smidge over 11k). However, what's of interest is the type of applications in which memcached is deployed. Memcached is most often used in web application to speed up page loads. Sites are almost² always dynamic and either have many clients (i.e. require horizontal scaling) or process piles of data (look to reduce processing time), or oftentimes both. This implies that the sites that use memcached contain more interesting info than simple static sites, and are an indicator of a potentially interesting site. Prominent users of memcached include LiveJournal (memcached was originally written by Brad Fitzpatrick for LJ), Wikipedia, Flickr, YouTube and Twitter.

I won't go into how memcached works, suffice it to say that since data tends to be read more often than written in common use cases the idea is to pre-render and store the finalised content inside the in-memory cache. When future requests ask for the page or data, it doesn't need to be regenerated but can be simply regurgitated from the cache. Their Wiki contains more background.

go-derper

We released go-derper, a tool for playing with memcached instances. It supports three basic modes of operations:

1. Fingerprinting memcacheds to determine interesting servers
2. Extracting a (user-limited) copy of the cache
3. Writing data into the cache

The tool has minor requirements: a recent Ruby and the memcache-client gem. What follows are basic use cases.

Fingerprinting

Let's assume you've scanned a hosting provider and found 239 potential targets using a basic .nse that hunts down open memcached instances³. You need to separate the wheat from the chaff and figure out which servers are potentially interesting; one way to do that is by extracting a bunch of metrics from each cache. Start small against one cache: insurrection:demo marco$ ruby go-derper.rb -f x.x.x.x [i] Scanning x.x.x.x x.x.x.x:11211 ============================== memcached 1.4.5 (1064) up 54:10:01:27, sys time Wed Aug 04 10:34:36 +0200 2010, utime=369388.17, stime=520925.98 Mem: Max 1024.00 MB, max item size = 1024.00 KB Network: curr conn 18, bytes read 44.69 TB, bytes written 695.93 GB Cache: get 514, set 93.41b, bytes stored 825.73 MB, curr item count 1.54m, total items 1.54m, total slabs 3 Stats capabilities: (stat) slabs settings items (set) (get)

44 terabytes read from the cache in 54 days with 1.5 million items stored? This cache is used quite frequently. There's an anomaly here in that the cache reports only 514 reads with 93 billion writes; however it's still worth exploring if only for the size.

We can run the same fingerprint scan against multiple hosts using

ruby go-derper.rb -f host1,host2,host3,...,hostn

or, if the hosts are in a file (one per line):

ruby go-derper.rb -F file_with_target_hosts

Output is either human-readable multiline (the default), or CSV. The latter helps for quickly rearranging and sorting the output to determine potential targets, and is enabled with the "-c" switch:

ruby go-derper.rb -c csv -f host1,host2,host3,...,hostn

Lastly, the monitor mode (-m) will loop forever while retrieving certain statistics and keep track of differences between iterations, in order to determine whether the cache appears to be in active use.

Mining

Once you've identified a potentially interesting target, it's time to mine that cache. The basic leach switch is "-l":

insurrection:demo marco$ ruby go-derper.rb -l -s x.x.x.x [w] No output directory specified, defaulting to ./output [w] No prefix supplied, using "run1"

This will extract data from the cache in the form of a key and its value, and save the value in a file under the "./output" directory by default (if this directory doesn't exist then the tool will exit so make sure it's present.) This means a separate file is created for every retrieved value. Output directories and file prefixes are adjustable with "-o" and "-r" respectively, however it's usually safe to leave these alone.

By default, go-derper fetches 10 keys per slab (see the memcached docs for a discussion on slabs; basically similar-sized entries are grouped together.) This default is intentionally low; on an actual assessment this could run into six figures. Use the "-K" switch to adjust:

ruby go-derper.rb -l -K 100 -s x.x.x.x

As mentioned, retrieved data is stored in the "./ouput" directory (or elsewhere if "-o" is used). Within this directory, each new run of the tool produces a set of files prefixed with "runN" in order to keep multiple runs separate. The files produced are:

• runN-index, an index file containing metadata about each entry retrieved
• runN-<md5>, a file containing the bytestream from a retrieved value

The mapping between key and file in which the value is stored occurs in the index file, which is useful in that potentially malicious data (keynames) aren't used when interacting with your local filesystem APIs.

At this point, there will (hopefully) be a large number of files in your output directory, which may contain useful info. Start grepping.

What we found with a bit of field experience was that mining large caches can take some time, and repeating grep gets quite boring. The tool permits you to supply your own set of regular expressions which will be applied to each retrieved value; matches are printed to the screen and this provides a scroll-by view of bits of data that may pique your interest (things like URLs, email addresses, session IDs, strings starting with "user", "pass" or "auth", cookies, IP addresses etc). The "-R" switch enables this feature and takes a file containing regexes as its sole argument:

ruby go-derper.rb -l -K 100 -R regexs.txt -s x.x.x.x

Over-writing

In this blog entry I don't cover the kinds of data we discovered (it'll be subject to a separate entry), however it may come to pass that you discover an interesting cache entry that you'd like to overwrite. Recall entries were stored in "./output" by default, with a prefix of "runN". If the interesting entry was stored in "output/run1-e94aae85bd3469d929727bee5009dddd", edit the file in whatever manner you see fit and save it to your local disk. Then, tell go-derper to write the entry back into the cache with:

ruby go-derper.rb -w output/run1-e94aae85bd3469d929727bee5009dddd

This syntax is simple since go-derper will figure out the target server and key from the run's index file.

And so?

Go-derper permits basic manipulations of a memcached instance. We haven't covered finding open instances or the kinds of data one may come across; these will be the subject of followup posts. Below are the slides from the talk, click through to SlideShare for the downloadable PDF. ¹ http://www.memcached.org

² We're hedging here, but we've not come across a static memcached site.

³ If so, you may be as surprised as we were in finding this many open instances.

Since joining SensePost I've had a chance to get down and dirty with the threat modeling tool. The original principle behind the tool, first released in 2007 at CSI NetSec, was to throw out existing threat modeling techniques (it's really attack-focused risk) and start from scratch. It's a good idea and the SensePost approach fits nicely between the heavily formalised models like Octave and the quick-n-dirty's like attack trees. It allows fairly simple modeling of the organisation/system to quickly produce an exponentially larger list of possible risks and rank them.

We've had some time and a few bits of practical work to enhance the tool and our thinking about it. At first, I thought it would need an overhaul, mostly because I didn't like the terminology (hat tip to Mr Bejtlich). But, in testament to Charl's original thinking & the flexibility of the tool, no significant changes to the code were required. We're happy to announce version 2.1 is now available at our new tools page. In addition, much of our exploration of other threat modeling techniques was converted into a workshop of which the slides are available (approx 30MB).

The majority of the changes were in the equation. The discussion below will give you a good idea of how you can play with the equation to fundamentally change how the tool works.

There are 5 values you can play with in the equation:

1. imp - the impact of a risk being realised
2. lik - the likelihood of the risk occurring
3. int - the value of an asset (represented by an interface to that asset)
4. usr & loc - the measurable trust placed in a user & location respectively

The current default formula is:

In English that translates to: The risk is equal to; the average of the impact of the attack and it's likelihood, combined with the value of the asset (exposed through a particular interface), and reduced by the trust of the user performing the attack and the location they are performing it from.

We felt there were two problems with this equation:

1. It doesn't acknowledge impact as linked to value. e.g. You can't have a huge impact on something of low value.
2. It doesn't see trust as linked to likelihood. e.g. a trusted user in a trusted location is less likely to commit an attack.
3. It double weights trust with location and user trust counting at full weight.
4. It's maybe a little far from semi-consensual views on the subject

After much internal wrangling, and some actual work on modeling fairly complex stuff, we came up with a new equation. While we feel this works better, it does mean the way things are modeled changes, and hence backwards compatibility with existing models is broken (but you don't need to use this equation). The new equation (consider the risk= implied) is:

Once again in English: The risk of an attack is; the likelihood of the attack reduced by the average of both the trust in the user & location, combined with, the value of the asset reduced by the potential impact of the attack (value at risk). (The 0.2 & 2.5 are just to make it fit the scales. Specifically, the 0.2 is because the scale of the entities is 1-5 and we're looking to make a percentage, and the 2.5 is to fit the 0-25 scale on the final graph.)

The key change which breaks backward compatibility here is that impact now becomes a moderator on value. i.e. the impact of an attack determines how much of the asset's value is exposed.

The way things are now modeled, interfaces represent the value of a system. For the most part, all a system's interfaces should have the same value, because as we often see, even minor interfaces that expose limited functionality can often be abused for a full compromise. However, the actual attack (called threats in the tool) determined how much of that value is exposed. For example, a worst-case XSS is (depending on the system of course) probably going to expose less of the system's value than a malicious sysadmin publicly pwning it (once again, dependent on the system and controls in place).

Unfortunately, there's still no provable way to perform threat modeling, but we feel we can go quite far in providing a quick and useful way of enumerating and prioritising attacks (and hence defenses) across complex system.

In a future blog post, I hope to cover some of the really cool scenario planning the tool can let you do, and the pretty graphs it gave us an excuse to justify budgets with.

[ Credit to the Online LaTeX Equation Editor for the formulas, although if you'd like to copy paste the formula described above into the tool, here's an ascii version:

( ( ( lik * ( ( ( (6 - usr) + (6 - loc) ) / 2 ) * 0.2 ) ) + ( int * ( imp * 0.2 ) ) ) * 2.5 )

]

The ITWeb security summit is coming up next week from the 11th to 13th of May. This is a conference we're quite excited about, and have been involved in for the last few years, but most recently, we've been able to further our involvement beyond just speaking.

For years I jealously watched as SensePost'ers would trundle all over the world shaking hands and drinking beer with the leet haxors of the world. Then a few years ago, the ITWeb Security Summit brought over Kevin Mitnick. I remember sitting in the audience awe'd not so much by what was said (sorry Kevin, I'm sure it was interesting) but at the fact a real celebrity hacker was meters from me. I still keep his lock-pick business card as a memento. Since then, the summit has gotten bigger and better. ITWeb previously brought out people like Bruce Schneier (who I think thought I was a stalker), David Litchfield, Johnny Long (he's African now), Johny Cache, Richard Stiennon, Roberto Preatoni and Phil Zimmerman (he video conf'ed in from his hospital bed after emergency heart surgery).

While meeting some of the international speakers was awesome, there was always a feeling that the conference was too vendor dominated. To help remedy this, last year SensePost was asked to put together a technical committee. SensePost's guidance on international speakers had an immediate effect and last year we had a ton of hacker rock stars: Jeremiah Grossman, Window Snyder, Adam Shostack, Mike Dahn, Tyler Moore, Frank Artes, Phil Zimmerman (this time IRL) and even The Gruq washed himself and made it over. In addition to the international speakers, the technical committee (which I was lucky enough to be part of) evaluated and voted on all talks, with the ability to vote out sponsor talks if they weren't up to scratch. While we had some teething problems (for example we weren't able to review all final presentations in detail) and made a mistake in trying to fit more speakers into a "turbo track", I feel the quality of the conference improved significantly.

After the conference, one of the awesome memories was the "Hackers on Safari" trip we took the international speakers on (and some of the technical committee, if they agreed to do dishes). It proved to be a really great way to "sell" South Africa to the international speakers. As we watched a battery of cameras synchronously snap many pictures of the "the asses of Africa" (the animals kept turning their back on us), we were reminded what a great place South Africa is.

This year is looking even better than last. There's a solid line up of international speakers: Kingpin, Moxie, Charlie Miller, FX, Dino Dai Zovi, Saumil Shah, Nitesh Dhanjani & Jeremiah Grossman. In addition, a third track has been created for security products with the other two focusing on the technical and business aspects of security respectively. We should see a lot of quality South African talks. Unfortunately, some promising talks and speakers had to be dropped to make space, but hopefully this is an indicator of higher quality and popularity rather than poor judgement.

Additionally, this year on the 13th of May @7pm (the last day of the conference) there is a hacker's party organised by our local unconference ZaCon (for full details follow the link), which is within walking distance from the conference venue. The party's aim is to raise funds for Hackers for Charity, with voluntary donations of R50 being asked, and HFC shirts for sale. Hopefully it will also provide a chance for members of the local scene who are unable to afford ITWeb tickets the ability to meet some of the international and local speakers.

Considering how freely i've ranted on our blog over the past few years i found it incredibly hard to to write this post. SensePost has been my home for the better part of a decade and i have been haroon@sensepost.com much more than i have been haroon meer.

In truly boring last post manner i wanted to quickly say thanks to everyone for making it such a fun ride. From the awesome people who took a chance on us when we were scarily young and foolish, to the guys (and girls) who joined us to help make SP elite. From the many customers who tolerated my sloppy dressing to Secure Data Holdings who have been awesome in every interaction we have ever had with them. From the people who have used our tools, read our work and contributed ideas to the people who read this blog (Hi Mom!).

Seriously.. thanks muchly!

It's been an awesome 10 years and with the quality of guys that remain at SensePost, it's a safe bet that the next 10 are going to be even better..

The question that everyone asks me is "what now?". The short answer still has 2 parts..

• I'm going to take a vacation.. (a short one, but im hoping to spend a week or 2 re-introducing myself to family members who vaguely recall me..)
• I'm going to be starting in a new direction, with [thinkst]

I won't go into tremendous detail here on thinkst (for that you will have to read/subscribe to my ramblings on http://blog.thinkst.com) - but the overarching hope is to focus slightly differently..

With Penetration Testing and Research over the past while I've spent a lot of time and energy trying to find new ways to break stuff, and new ways to break into stuff.. (it's been incredibly fun!)

I'm hoping now to be able to aim the same sort of bull-headedness at defending stuff, and at building solutions that give applications and networks a fighting chance.

I'll still pop in occasionally at the SensePost offices (mainly to have the coffee and lose at foosball), and my relationship with Secure Data Holdings also remains intact (Other than our historical relationship, Thinkst is doing some consulting work for SDH, making them our first customer!). Hey.. you might even still find me bending your ear on this blog..

So.. all that remains is to say thanks again.. it's been amazingly fun, incredibly rewarding and "rockingly leet"

Sincerely

/mh

After ten fascinating years, during which many people have contributed in so many ways to the place that is SensePost, by strange coincidence it falls on me to pen the words that mark our first decade in existence. To quote Robert Hunter: "What a long strange trip it's been". SensePost was officially founded on February 14, 2000. Of everyone who was involved at that time, I'm the only one still working here, which earns me the dubious honor of 'oldest employee'. Do I get a gold watch? I meant to think much more over the last few weeks and months about how we should celebrate this day, or what I would write in a letter like this, but in the end (business being business) I'm writing this in a rush on a Sunday evening, with another three big things to complete before I allow myself to go to bed. Then again much of our success (in so far as we've been a success) happened in hurry on a Sunday night, so let's not write this little piece off too soon, shall we?

The vision for SensePost developed between myself and Roelof Temmingh late in 1999. To be fair, Roelof was by far the more skilled and experienced at that time, and the notion of a commercial venture rooted in computer hacking as a service was born primarily with him. But I like to think I played a small part in shaping and molding the ideas that formed during the early part of that summer. Certainly I believe it was my epiphany that as long as we waited for others to make the calls, we would never never really be in charge of our own destiny, that finally convinced us to leave our jobs and set out on this new venture. It was the height of the 'dotcom' boom, we knew more about everything than anyone, and we thought we'd be rich before two years were out. Of course it wasn't that simple, but its been a crazy happy journey nevertheless and I don't regret a minute of it.

It wasn't all about money of course. There was also a dream. We saw a small group of people, technical, hard working, passionate about computers and security, and with poor fashion sense. We had wild ideas about a grunge-style internet cafe with drinks named after shell commands, big screens and 70's pop. I also recall some discussions about a scooter with a fax machine mounted on it, but we won't go there. Basically, we had no idea what we were doing. Yup. Roelof and I had passion, idealism, energy, a whole lot of arrogance, and a little bit of skill, but not much more. We were 24 years old, had about US$ 6,000 between us, and probably barely enough collective business acumen to open a cheque account.

Help came from a very unexpected place. As it turned out the managing director of the company we were leaving, an ueber-suite, the boss of our boss, public enemy number one, prime-evil himself, had resigned the company just weeks before we did. His name is Luc de Graeve and instead of calling down the gods of corporate South Africa to punish us for our insolence, he kindly and gently offered us advice and support, which we eventually, suspiciously, accepted. And so was formed a relationship that would culminate with Luc becoming a major shareholder and our managing director for eight years until after we eventually sold to Secure Data in 2008.

In the sidelines at that time, but a secret member of our troupe right from the start, was Chris Erasmus. Chris has joined a team Roelof was starting at our previous company and we promised to invite him in the moment SensePost was on its feet. And so Chris joined us as a shareholder only a few short months after we started. Although Chris was the first of the founders to leave, he played a formative role in establishing our culture, values and identify. His sincere manner and unique stye left an indelible impression on each of us and on the business itself that can still be felt today.

And then there was Jaco. Jaco van Graan had also worked with Roelof, Luc and me, but had left before the rest of us to take a security job at a major ISP. On the side, he and two friends had started an accounting and audit practice called TJC. They planned to specialize in helping small businesses like ours and approached us with a very attractive proposal. Before too long Jaco would join us as 'financial director' and BS 7799 specialist. We wondered at the time whether it wasn't too soon to require a full time financial manager, but the indisputable balance and control we've had in all our financial and commercial matters since that day testify that it was the right call.

Next join our team was Haroon Meer. We met him online while he worked at Durban university and invited him to come visit us at the 'office' we ran out of Roelof's master bedroom. He soon went on to join the directors and eventually become our technical director and in many ways the heart and soul of our business. After I finish writing this post, I have to write some words for his farewell. His contract with Secure Data has expired and he's moving on to his next big adventure. I sincerely wish him well, but already miss him dearly.

The contract I'm referring to with Secure Data is part of the purchase agreement with them. Under that agreement three of the shareholders - myself, Haroon and Jaco - were obliged to stay for a fixed term after the purchase. That period has not yet ended, but Secure Data has allowed for him to break a little early. In this, and many other things, Secure Data has been a good partner to us. The decision to sell the business back in 2008 was a not an easy one and we entered into the deal and subsequent contract period with more than a little trepidation. But Dean and Johan have understood us well and have graciously allowed us to continue being who we are. Thus, I say with confidence, that nothing has changed in our culture or values since joining Secure Data. I suspect this is unusual in such cases, and I'm extremely grateful for it. Indeed, Dean has proven to be wise and insightful leader.

So our tenth birthday also marks the end of our journey with Haroon. Of the original people, only myself and Jaco now remain. I feel I've said goodbye to too many people over the past decade. I hate it. But I've also come to learn that the business is bigger than any individual one of us. Each time somebody leaves I dread it, and each time we somehow survive. Over the years the business has grown from strength to strength and today we boast much more skill, energy and talent than Roelof, Haroon, Chris, Luc, Jaco or I ever had.

Time doesn't allow me to tell the whole SensePost story in detail and I guess there's really not all that much to tell. But there are some players I just have to mention: My deepest love and respect to Roelof - my friend and mentor - and Luc - long our leader and the biggest set of footsteps anyone ever had to follow. @haroonmeer - I've already said how much I'll miss you. Chris - I hope to see you again soon. Kim, Gareth, Lizelle, Christoff, Herman, Jacof, Nithen, George, BradleyW, Craig, Lohan, Frank, James, Glenn - thank you all sharing a part of your journeys with us. And to our customers: I can't mention you by name, but some of you have supported us from the very beginning, and all of you have been gracious, patient, loyal and extremely supportive. Thank you! Without you we would lack any meaning. And I must mention … Black Hat. Ping and Jeff gave us a chance when nobody had to, and opened up the door that would eventually allow us to become a truly global company with customers on all five continents. Thank you Ping and Jeff. My hope is only that we can give people the kind of leg-up that Black Hat gave us.

So how have we done over the last ten years? The other day Haroon - ever our conscience - mentioned Sun CEO Jon Schwartz's memo at the time of the acquisition by Oracle. Haroon was saying how he kept record of the memo to remind himself of the kind of company he wants to work for, so I thought it might offer a good benchmark against which we can judge ourselves…

Schwartz: "Sun's people have always stood apart as the brightest, most passionate, and most inspiring… I've always been surrounded by the best and brightest individuals I've ever come across…"

I certainly don't count myself amongst the best and the brightest, and SensePost is certainly no Sun, but I can say honestly and sincerely, in the words of Schwartz himself: It's "been an honor and privilege, for which I'm enormously thankful".

Schwartz: "[Our] Technology, alongside our employees and partners, have changed the world"

From the beginning, SensePost has had the courage to build and release technologies that make a difference to how we think and work, have made a difference to our industry and ultimately to our customers. And we're still doing it today. Sure, our's is a small galaxy, but I'm proud of the difference we've made in it.

Schwartz: "Amidst the toughest market and customer situations imaginable, I'm proud we've always acted with integrity, with a sense for what's right, and not simply what's expedient."

This is perhaps the part of our makeup of which I'm the most proud. SensePost has always been a values-driven organization and I believe I can say with all truth that we've never compromised on our values. We've been fair and honest in all our dealings with our customers, our staff, our suppliers and even our competitors. I'm proud to say that I can't think of one person in our industry, in South Africa or abroad, that I'd be ashamed to run into.

Much of what's happened over the last ten years has taken me by surprise, so its hard to comment intelligently on what the next ten years will hold. But what I do know is this: At its heart, I believe, SensePost is about learning. Learning and teaching. We believed at the time (arrogantly I suppose) that we knew more than anyone else. Not anyone else in the whole world I mean, but the more than the people and businesses we were dealing with at the time. And our heart… was to teach them.

This spirit of teaching is still at the heart of our business model, and must remain at our own hearts also. Teaching is how we add value to everyone we deal with - our staff, but most especially our customers. Its a generous spirit, for to teach is a fundamentally generous thing. Teaching is not about fame or money, its about sincerely caring for the other and wanting to empower and enable them. The fame and money, if you're lucky, will follow.

To be a good teacher, however, one must first be a student. Thus, as the rate of technological development catapults, and as the world around us becomes ever more complex, we need to learn. We need to hunger for knowledge, insight and understanding and seek it out at every cost. We need to work harder, think deeper, push ourselves at every opportunity. The moment we stop doing this. The moment we start to make assumptions and take things for granted… that will be the moment when we start to fail.

And to end, two more quotes from Schwartz:

"We're known as self-starters, capable of ethically managing through complexity and change, for delivering when called upon, and for inventing and building the future. With the world economy stabilizing, I'm very confident you'll land on your feet. You're a talented, tenacious group, and there's always opportunity for great people."

So, to Jaco's team in finance - thank you for keeping the wheels turning and for reminding us what it is to 'serve' others. To the analysts in our assessment team - thank you for the continuous quality and passion of your work. That's how we roll. To the VMS team and developers, you hold the keys to our future. Keep it up - your moment will soon come. To Shane and Bradley, sales and presales - you are our link to our customers and the rudder that steers our ship. To Dominic in consulting - thanks for joining us at last. To Junaid ... welcome on board. May your full potentials be realized with us. To others that have already left us - thank you for sharing with us - may you have success wherever your paths have taken you.

And finally:

"Thank you, again, for the privilege and honor of working together."

URL for Schwartz's memo to Sun: http://news.cnet.com/8301-1001_3-10440125-92.html