Groovy Monkey and Syntax Checking

Groovy Monkey is an Eclipse plugin which allows you to quickly script plugins for Eclipse. While I'm not that interested in making a whole bunch of plugins for Eclipse, I can see how this can be useful. I've found it, so far, to be very handy for doing quick syntax checks on code. The better static analysis tools out there do data flow analysis to reduce the potential of false positives (er - non-exploitables), but I like to be a lot more strict. There are some things that almost always end up being dangerous, but those constructs won't end up in a static analysis. If you're trying to convince people to always use <c:out />, then <%= shouldn't be used. So just messing with the example script gives me the ability to add that little check.

Next I'll add in checks for potentially bad data access mechanisms (createStatement, executeQuery(String), etc.) just for flagging for the developer to keep an eye on. The beauty of this is that the results end up in My Tasks instead of some separate perspective.

And yes, for anything but the <%=, I could just write a semantic rule, but this will end up in my task list, and is generally far quicker.

I've Said it Before, but...

Now, ordinarily, I hate screen-scraping. If there's any other way to get the raw data, I go there first. I go through whatever (ethical) channels necessary to get direct access to the source data, whether it be in a relational database, LDAP, XML, straight text, spreadsheets, or made up out of nowhere. I can't stand screen-scraping because screen-scraping is normally very sensitive to change. Screen-scraping HTML is not generally as bad as telnet or green-screen, but it's still bad enough that I try to avoid doing it - particularly when the HTML is malformed.

But today was another occasion where it was necessary simply because I couldn't get access to the systems I needed in the timeframe I needed. What would have made this one more difficult is that the source HTML had very few line breaks to do text parsing, and the HTML was also not properly XML formatted, and to boot, was poorly-enough HTML formatted that an event-based HTML parser simply wasn't going to work out.

Fortunately, I've done a little screen scraping with Groovy before, and this task wasn't significantly more difficult than other tasks I've done before. And again, NekoHTML came to the rescue. NekoHTML takes poorly-formatted HTML code (in this case, really poorly formatted) and balances unbalanced tags (had plenty), closes unclosed tags (had many), quotes unquoted attribute values (lots of those today), and gives sensible default values to un-valued attributes (and a bunch of those, too). What results is actually well-formed (not necessarily validating, but well-formed) XML, which you can parse with any ordinary XML parse.

In this case, I used XmlParser, which allows me to do very nice GPath queries. GPath works similarly to XPath, but allows you to find really complicated paths. For example, in English, "find me the text in all the <strong< tags that are under <a> tags such that their 'href' attribute matches this regular expression." In an event-based parser, that would take a lot of work, in DOM it would be easier, but still a lot of code, and the XPath would just be nasty. In GPath, it looks like this:

texts = page.depthFirst().A.grep { it.'@href' =~ /^.*\.action\?foo=(.*)$/ }.collect { it.value.STRONG.value }

Which is much fewer lines of code.

Now, what does this have to do with application security? For those who do black-box testing, there are times that your toolkit doesn't quite have enough in it. Your proxy is powerful, but just won't get you all the values you need. If there are special considerations you need to take in order to try brute-force authentication, or if you've found a good SQL injection attack, but the way the data comes back is finicky, scripting is often appropriate. So if you're looking for another swiss-army knife, some (understandably) are still Perl enthusiasts, (understandably) happy with Python, (understandably) infatuated with Ruby, but so far, Groovy has really been doing good work for me.

That being said, the GPath statements aren't specific to HTML - GPath works with XML, which is why you might need NekoHTML. And NekoHTML isn't specific to Groovy - it's a java library, so you can use it with your other java code and use whatever XML handling you prefer.

Adding a Request Token for RemoteField

I've been writing a 15 minute expense tracker in Grails, and it was done in 8 minutes, but I've been spending little snippets of time improving it since that initial 8 minutes. After some of the discussions at BlackHat, I decided I'd try to start making some posts on how to do some things more safely in Grails.
The RemoteField Tag in Grails is uber-easy to use, you give it something like the following:

<g:remoteField name="description"
value="${expense?.description}"
action="updateDescription"
id="${expense?.id}"
update="resultdiv">


The problem is that with all the default documentation, there are two problems with this - key exposure, so you have to check this user's permission to edit the specified object, and XSRF. This isn't a problem with the tag, it's a problem with the easy way of doing things in Grails - and this is identical to Rails. An ordinary CRUD call in [Gr|R]ails makes a URL like http://host/<app>/<controller>/<action>/<id>, and the ID is available as params.id. And that ID is (under normal scaffolding), the primary key of the domain object in question. So an attacker just needs to put on their site an iframe that makes the user do something like http://victim/app/epxense/delete/382, and if there's no permission checking, it goes bye-bye.

Adding a level of indirection to keep the PK in the session and then meaningless ID's on the URL won't solve this because an attacker can just make a user delete all their own entries, etc. So you have to use a token that proves the user visited the page first, and making the user solve a new CAPTCHA for every keyup is silly, right?
So generally, what you see is a hidden form element with a long random token that is also stored on the session. When the form goes back, the two tokens are compared to (somewhat) guarantee that the user was on the correct "setup" page before coming to the submission form.

I started down a handful of paths, and the following were less than elegant:

• Altering the remoteField itself was less than ideal because you would actually have to add several attributes to the tag to determine how you wanted to deal with the token - do you want to use the same token for all elements on the page? Do you want to use a different token for each element? What do you want to name it? Those decisions have to be carried across to the action that gets called by the remoteField as well.
• Actually replacing the id attribute with the session token actually worked, but only for one field on one form. If you wanted to do this in several places, because you've abstracted the id out, you'd have to make changes to a lot of existing scaffolding to get it right.
• One thing I didn't try was keeping the id's from a list() closure in session, keyed by new tokens generated for each one. This would take quite a bit of work, but would also remove some key exposure at the same time - as long as you don't use the tokens in the session to order by. This would also allow you to add token checking as part of a beforeInterceptor because then the keys are the token ID's.

So I ended up with two methods that ultimately worked pretty well.

The first method is to use the paramName attribute of the remoteField tag. If you don't specify the paramName, the value that gets passed back will be with the request attribute "value". So the post parameter looks like "value=The+new+value". When I first put the tag together, I had no compelling reason to use anything other than "value" as the name, so I used this to hold the token:

<remoteField name="description"
value="${expense?.description}"
id="${expense?.id}"
update="statusdiv"
paramName=${token}">

Then in the controller:

if (! params[session.token]) {
render('Bad token')
} else {
...


The second method I tried was to append the token as another request parameter in the id attribute, like so:
id="${expense?.id + '?token=' + token}"

This also works, although it muttles up the tag a bit, but it makes more sense in the controller - just compare session.token to params.token.

Unfortunately, it was still up to me to generate the token, remember to put it in the form fields, and to remember to check it on the way back in. Not perfect, but security always comes at somebody's cost. In this case, it was mine.

Going off the Rails on a Groovy Train - Part 2

Well, in Part 1, I promised to explain why I went back to Groovy after ignoring it for so long. But to explain that would be to explain a whole bunch of other stuff. While not completely security related, it might be interesting for developers to see the thought process that makes a wishy-washy person like me jump ships on languages.

First, I'm not a religious zealot about languages. To me, languages are just tools, and learning a new language is a matter of learning syntax, and how to find the libraries I need to do the job.

For the longest time, I was a Perl coder - er....line noise coder. My code was write-only. In Java parlance, write once, ignore forevermore. The two major benefits of perl for me were availability of libraries and documentation for those libraries (owing to perl's maturity), and syntactical sugar. (Of course, I didn't know the benefits of the second until I couldn't use it anymore).

I used perl a lot. But for pen testing, I kept coming into situations where it wouldn't cut it anymore - almost always due to finicky proxy requirements. I could do SSL, I could do proxy, I could do NTLM autentication, but I always seemed to have issues dealing with combinations of the three. And it seemed that many, many pen tests had something come up that required some scripting and that the tools out there simply couldn't handle.

So for one assessment I tried out Commons HttpClient with Beanshell. I had been using Java and Struts for web development for awhile, so the transition to Beanshell was natural. And HttpClient did exactly what I needed when I couldn't seem to make Perl do it. (My apologies to the perl people for giving up so easily, but I had an immediate need. And my apologies to the Python people because I prolly never gave Python a fair shake).

So I stuck with beanshell and HttpClient for awhile. But beanshell never really made me happy. While a dynamically-typed java was nicer, it never got to feel like a scripting language because I still didn't have a lot of syntactical sugar. My colleagues jumped on the Ruby bandwagon quickly, and I jumped off as quickly because the two things I needed to be able to do with it were difficult to find libraries for. And in fact, did a lot of assessment work where I'd use beanshell to do the http plumbing, and perl to do the parsing because parsing HTML in Java is so un-natural - possible, but not natural.

I'm not sure why I gave up on groovy so long ago. I'm not sure why that one afternoon I dusted off beanshell instead of dusting off groovy, but I wish I had picked groovy then. A couple of weeks ago, I decided to look at groovy again when I had had enough with beanshell, and I was floored. It was like magic. All the library maturity of Java (I still get to use Jakarta stuff) with all the syntactical sugar of perl and all the readability of Ruby. And that there are builders for almost anything hierarchical in nature.