A friend recently asked if I would post about using md5deep; he was trying to figure out how to use it to compare against the NSRL hash library and was having some difficulties. Well, maybe it's stretching to say he asked if I would. He asked for some help in figuring it out, and at the end of it all, I offered to post some stuff; he said it might be useful. So if I put a PR spin on it, I was specifically asked, because of my great and wondrous knowledge. How's that? :)
Oh, and if you're wondering at my use of the term "friend," you can always substitute "person who pretends not to hate me as much as others do" if that makes you feel better. I know some of you don't think it's possible that I actually have friends, so I feel I have to give you a way out...
Anyhow, to be honest, I've not ever used md5deep for that specific purpose. I've always used something like FTK or EnCase. I know, I know, don't be hatin'. I do a lot with command line tools, and prefer them in many - if not most - cases, but there are some instances where (completely dependent on the scenario), a GUI is just easier and/or the better option. And for what I've done with the NSRL hash libraries, those tools were what was called for (there was more than just deNISTing at hand).
That said, the way I've used md5deep should still accommodate comparing against the NSRL hashes, with just a little bit of tweaking. I've always used it to compare and verify files being copied from various data sources (typically network shares), when using a copy application that couldn't do so. So in that scenario, it works out something like this:
We want recursion, possibly a progress/status indicator, and an output file that can be used as the input file for hash comparison.
Hash the source like:
user> md5deep -re "g:loose_PST_files" > f:psthash.txt
Hash the destination (and compare to hash of source):
user> md5deep -renx f:psthash.txt "f:loose_PST_files" > f:psterrorhash.txt
This breaks down as follows:
-r = recursion
-e = progress/status
-n = displays names of files not matching the input file
-x = negative hash matching; reports files not in the input file g: is the mapped or local source drive f: is the destination drive
Order for source is: > # the flags have to be combined together (ie, -re) or it errors. It will automatically output to stdout, so we have to redirect to logfile; this gives filename and hash value only (other format cannot be read properly for use as input file).
Order for destination is: > # the error log immediately follows the flag as the first variable; this is read and compared against the hashes of the destination. Any errors in the comparison (whether the file is there but shouldn't be, or isn't there but should be) will output to the error log. The size of the log should be 0 KB when all is done, and (obviously) empty - thus showing there were no errors/mismatches.
These were notes I made in the past, and I think were most recently updated with version 3.4. The tool is now on 4.3, so I don't know if the following has changed, but ... as noted above, the md5deep comparison (input) file had to have hash value and filename only, or could not be properly read. I couldn't find anything in the documentation to say one way or the other, but it's possible that the NSRL lists might be problematic, even though md5deep is only looking at the hash and not the filename/location. I did have occasion recently to verify the integrity of some evidence migrated to new storage, and ran md5deep in the manner above; it worked great. The only thing I'll note is that if you do want the progress indicator, it WILL slow down the process; use with discretion. You have been warned.
There were a couple posts related to using NSRLSrvr and NSRLLookup that showed up on the radar. The first I passed along, and the second my friend found on his own; apparently it was helpful. I don't know the first author, but Patrick puts out some good stuff on Sysforensics. If you don't keep tabs on his blog, you should. So without further ado, here they are, and they should work just fine to use md5deep to compare against NSRL.
And this wouldn't be complete if I didn't mention the NSRLookup service from Kyrus Tech:
Which I first saw mentioned last year on Jesse Kornblum's site:
Have fun and happy hunting (or, uh, happy hashing)!