I'd like to pick up on one or two comments from an earlier thread and bring the subject of report standardization into the spotlight.
This is a subject area which has cropped up before (in these forums and elsewhere) and also one which has given me pause for thought in practice - in common with most of us here, I imagine. I think the time is right to give some serious consideration as to whether the standard of reporting delivered by computer forensics practitioners is all that it could be and, more specifically, is the introduction of a suitably structured and widely accepted model a worthwhile goal to aim for.
A number of benefits have already been suggested for such a model, some of these being increased efficiency, increased accuracy, improvements in communicating with other parties and an increase in professional credibility. In addition, two paths have been suggested for achieving this goal - one, get the major computer forensic groups and organisations to agree on such a model and push it out to their members, the other, develop a model at a grass roots level and grow support and acceptance for it amongst members of the various computer forensics forums.
I'd like to request further comments from all of us here. Do you think there's anything wrong in principle with a standardized reporting model? If not, could such a model be developed which serves to provide the benefits mentioned above without undue restriction being placed on the report writer? What would be the best way of creating such a model? Would the time and effort spent developing a suitable model be worthwhile?
All thoughts welcome!
I'd like to pick up on one or two comments from an earlier thread and bring the subject of report standardization into the spotlight.
This is a subject area which has cropped up before (in these forums and elsewhere) and also one which has given me pause for thought in practice - in common with most of us here, I imagine. I think the time is right to give some serious consideration as to whether the standard of reporting delivered by computer forensics practitioners is all that it could be and, more specifically, is the introduction of a suitably structured and widely accepted model a worthwhile goal to aim for.
A number of benefits have already been suggested for such a model, some of these being increased efficiency, increased accuracy, improvements in communicating with other parties and an increase in professional credibility. In addition, two paths have been suggested for achieving this goal - one, get the major computer forensic groups and organisations to agree on such a model and push it out to their members, the other, develop a model at a grass roots level and grow support and acceptance for it amongst members of the various computer forensics forums.
I'd like to request further comments from all of us here. Do you think there's anything wrong in principle with a standardized reporting model? If not, could such a model be developed which serves to provide the benefits mentioned above without undue restriction being placed on the report writer? What would be the best way of creating such a model? Would the time and effort spent developing a suitable model be worthwhile?
All thought welcome!
Jamie before jumping in with suggestions of what should/could be contained in a Model Report may I just add a viewpoint from someone in the UK
Model reports have been put forward in various styles for quite sometime, by way of illustration, the Law Society Directory of Expert Witness set down a template in their Directory, which was helpful and informative. But what might work for an Architectural Specialist may not be helpful to a Computer Specialist.
The Academy of Experts produced their own Model Report
http//
"The Model form of Expert’s Report was developed by the Academy’s Judicial Committee (a committee of senior members of the Judiciary, chaired by a Law Lord) to provide Experts with a ‘judge friendly’ format which would be accepted by the courts."
A copy (.pdf document) of content for expert reports can be downloaded here
Model Reports have also been identified by the UK Register of Experts
http//
Whilst these guides are helpful, rarely have I seen two expert reports use a common template.
If the Report is not for expert purposes but client-specific purposes then where would a Model Report (if not Expert Report) be used and for what purpose? By that I mean, would the client report expect to contain general statements
Details of Author
Qualification & Experience
Purpose of Report
Tasks involved for work
Information Assessed for Tasks
Results of Tasks
Conclusion
or technology specific statements??? For example a number of examination systems have built in standard template reports based upon the extraction and harvesting of data the system performed.
I hope you will see my comments as simply queries in order that I know how to respond to your question/s in this thread and not intended to suggest your objective cannot be achieved.
I hope you will see my comments as simply queries in order that I know how to respond to your question/s in this thread and not intended to suggest your objective cannot be achieved.
Oh, absolutely, and I should stress that while my own feeling is that there is room for considerable improvement here - perhaps just by formalising a few very basic things - there's certainly the potential to disappear down the rabbit hole if we're not clear about what we're trying to achieve (and for whose benefit). It's also tremendously useful to look at what attempts have been made in the past to achieve similar objectives and discuss how successful (or not) they've been. It's a very useful post and I'll be taking a look at the reports shortly.
Just a couple of thoughts
I have always thought that one of the key skills that we have to develop is to target the content and style of report towards the "end user". The report I may write for one client may be very different in content, style, layout etc compared to another. At some basic level, you have to consider the requirements both of the person who will be reading (and hopefully understanding) the report, whilst at the same time, complying wth any specific requirements that are needed within the enviroment it is being used.
Secondly, be coming up with and promoting a template, there is a danger of doing "forensics by numbers" raher than treating each investigation on it's own merits and giving genuine consideration to the best way to approach the challenge and how to report the findings. Yes, this will produce conformity but there must be a danger of just becoming lazy and filling in the blanks for every report produced.
As usual, just my 2 penneth,
A number of benefits have already been suggested for such a model, some of these being increased efficiency, increased accuracy, improvements in communicating with other parties and an increase in professional credibility.
Who's "professional credibility"?
Why should I be investing my time and effort into increasing the professional credibility of either someone I don't know, or worse…a competitor?
In addition, two paths have been suggested for achieving this goal - one, get the major computer forensic groups and organisations to agree on such a model and push it out to their members, the other, develop a model at a grass roots level and grow support and acceptance for it amongst members of the various computer forensics forums.
I'm not sure this is going to work…there's just too many different environments and end-user customers for this to really work in any meaningful way.
I'd like to request further comments from all of us here. Do you think there's anything wrong in principle with a standardized reporting model?
Yes, I do.
Within the group I work in, we're standardizing our reporting…it only makes sense to do so. However, we've also seen reports from competitors…not often, but every now and then. If my report template is now being used by a competitor, then where is the business differentiator? What sets my team apart from a competitor?
Tools are fairly stagnant across the landscape, with the notable exception of any home-grown, in-house tools. Where competitive advantage lies is within the techniques of how those tools are used and the analysis that they are actually used for. Having a template that spells out what needs to be done levels the playing field and makes it even more difficult to differentiate the provided service, even beyond the marketing hype.
I have always thought that one of the key skills that we have to develop is to target the content and style of report towards the "end user". The report I may write for one client may be very different in content, style, layout etc compared to another. At some basic level, you have to consider the requirements both of the person who will be reading (and hopefully understanding) the report, whilst at the same time, complying wth any specific requirements that are needed within the enviroment it is being used.
No argument from me on that score (apart from some discussion around "layout" perhaps). I think what I'm talking about in terms of standardization is one level "above" the content and style of the report and geared more towards the most basic elements of any reporting system as far as they refer to the examiner and investigation itself i.e. who, why, what, when, where etc. To clarify, I'm referring to details of the examiner and parties involved, the purpose of the investigation, details of hardware and software and so on, not suggesting that we draw up somewhat restrictive guidelines for how procedures, results or conclusions are reported (other than defining specific areas within a report for those particular sections).
Secondly, be coming up with and promoting a template, there is a danger of doing "forensics by numbers" raher than treating each investigation on it's own merits and giving genuine consideration to the best way to approach the challenge and how to report the findings. Yes, this will produce conformity but there must be a danger of just becoming lazy and filling in the blanks for every report produced.
I suspect we're probably using different definitions of a template or model here but it's still a very useful discussion to have, primarily because I think this is likely to be used as an argument against standardization even though it's based on a misunderstanding (and clearly the onus is on anyone calling for greater standardization to explain why). I fully agree that any model which prevented or hindered an examiner from treating each investigation on its own merits is likely to fail, and so should it. While the basic principles of good practice are widely agreed upon we all know that each case is different from the next. I want to stress, though, that I'm not suggesting that we create a new model for procedural issues (I think there's a strong argument that guidelines based on fundamental principles are the best way to promote best practice while still allowing flexibility) but simply a standard way of reporting the basic elements of any digital evidence based investigation.
Basically, I'm referring to all those elements which are already included in reports "as standard" but which are reported in somewhat different fashion from one organisation to the next, even if those differences might simply be described as cosmetic (e.g. related to order or placement within the report). Standard reporting templates are already used by individual organisations, the question is - is it worthwhile attempting to bring those models together in order to achieve some of the benefits previously suggested? Or better, why is it not worthwhile attempting to do so?
Who's "professional credibility"?
I'm tempted to say "ours", if that's not too glib a response?
Why should I be investing my time and effort into increasing the professional credibility of either someone I don't know, or worse…a competitor?
Fair question, I suppose the answer lies in whether your motives are purely commercial/competitive or whether you see professionalism within your industry (in this case as evidenced by making an effort to improve communication with other professions) as a worthwhile goal in itself.
I'm not sure this is going to work…there's just too many different environments and end-user customers for this to really work in any meaningful way.
Quite possibly so, and if the clear conclusion after discussion is that that's the case then, I feel, it will have been a worthwhile exercise. With that said, it's not clear to me why different environments or end-users would be a prohibitive factor in standardizing/formalizing a very basic reporting framework.
Within the group I work in, we're standardizing our reporting…it only makes sense to do so. However, we've also seen reports from competitors…not often, but every now and then. If my report template is now being used by a competitor, then where is the business differentiator? What sets my team apart from a competitor?
The answer to that would presumably be related to the content of the report rather than the framework?
Having a template that spells out what needs to be done levels the playing field and makes it even more difficult to differentiate the provided service, even beyond the marketing hype.
I think you're overstating the case for the type of template I have in mind. Not to repeat my previous post but I'm mainly suggesting the standardization of some aspects (e.g. location) of elements which are already included within every report produced. I apologise if I've been unclear in that respect (although again, others may have different goals in mind when talking about standardization).
As more than 90% of my work is with CPA/CFEs I firmly believe standardization is very important to computer forensics. Part of the key to the scientific process is the ability of others to re-create our work. When a CPA reviews an audit performed by another auditor, the order of items in the audit opinion is the same. If something is missing or awry or lacking it is not hidden or buried, although if you have read an audit opinion recently it may not seem that way.
I can assure pbeardmore that there is no such thing as auditing by numbers, well there is but not in the way you reference "forensics by numbers". And Harlan to address your concerns about reports being a business differentiator, even though audits and the resultant audit opinions have certain requirements including content and formatting, I can assure you there are great differences in quality. Just because an audit opinion has certain requirements, the ability of the auditor to present the information is still quite important and is part of what sets one apart from another.
Another difference is in the quality of the work performed. Training and specialization of the auditor makes a great difference in how an audit is conducted and reported. For example if an auditor is versed in the banking industry, but picked up a not-for-profit audit, another auditor that specializes in not-for-profit will be able to tell. I believe the same is true in computer forensics. I do not work IIOC cases because I do not follow the whole chat room, IRC, whatever other things go into those cases. That is not to say I cannot examine a computer in question, I just do not believe that I would be providing the best service to my client by taking on that type of case.
To break from the audit opinion example, the reams of papers filed by attorneys with the courts also require certain levels of formatting and content to fit the requirements prescribed by law. And a quick read of those papers will certainly reveal differences in the abilities of those filing the papers.
As Harlan mentioned CF tools are relatively stagnant. In financial auditing, tools are probably even more stagnant. It is all about the skills of those wielding the tools and their investigative abilities that yield results. Additionally their ability to explain their results in the framework of the audit opinion standards sets the mediocre apart from the skilled.
In regard to professional credibility, States are beginning to regulate CF examiners as Private Detectives and I believe it is important that we as an industry begin to regulate ourselves and our work before we have regulation thrust upon us.
I have seen some problems with standard template reports which if you know the groups that regularly produce the report you know straightaway they use templates. The problem I find with these templates is that the author becomes lazy to the extent that
1) leaves out dated quotes in the template
2) inability to show how the general statements in the template actually has any relevance to the stated conclusions
3) to use general statements to identify why the author has not done something or not obtained certain materials yet the author has in the past for example used the same materials or even in a present case use such materials which the author has denegrated by way of the general statements
So maybe it is not a good idea to have Model Reports containing general statements as a part of the template, but that doesn't mean to say the Model Report can't have "Headings" as standard for a template, examples of which can be seen in the references I gave in my first post above.
I can assure pbeardmore that there is no such thing as auditing by numbers, well there is but not in the way you reference "forensics by numbers".
I think that although "auditing by numbers" is something which can obviously restrict the ingenuity and deviousness of an auditor in finding security flaws, there are a number of existing frameworks, not least the Open Source Security Testing Methodology Manual (OSSTMM) which, without proscribing method, describes key areas to examine and a reporting structure ( ISECOM, the producer, also offers a certification of report to ensure quality - at cost of course 😉 ). Beyond the OSSTMM, there are plenty of other "standards" related to IT Security that one can work from/to in order to structure an audit - ISO27001 for example - the ISACA also have guidelines, and the "Orange Book" has been around since 1983 ! (http//
Within the group I work in, we're standardizing our reporting…it only makes sense to do so. However, we've also seen reports from competitors…not often, but every now and then. If my report template is now being used by a competitor, then where is the business differentiator? What sets my team apart from a competitor?
The answer to that would presumably be related to the content of the report rather than the framework?
Absolutely, we all work to standards or frameworks all the time, be it ISO27001, OSSTMM or our own in house reporting standards - however, at the end of the day - some people will always be more efficient, comprehensive, articulate and experienced than others.
I guess that in some respects it levels the playing field a little, on the other hand it means that all of a sudden _content_ is king, and that just because a company has a nice font, a corporate image and a flashy report doesn't make the information better than a one man specialist.
Just because we choose to report in a unified _style_ doesn't mean that two examiners will come to the same conclusions, just that it will be clearer to identify where those differences of opinion are to a lay reader.
Picture it differently, you are being asked, as a juror, to understand the relevance of a piece of physical forensic evidence, say a chemical trace. You are presented with two conflicting reports, presented in totally different formats - how are you going to extract the key points from that ? Yet if there are two reports that are formatted in such a way as to show clearly where the opinions diverge - one can make a much clearer decision as to the relevance of any given finding.
I've argued for standardisation before, and found that whilst there is a lot of lip service to it, there is little to no actual real support in putting it together. I know that there are many constraints on people regarding time, availability, contractual constraints etc.
However, if this is changing, then that, in my opinion, is a great thing - and we should take advantage of it.
One of the hardest things that I found as student of Forensics, is trying to develop a reporting style. I could find data, I could attribute it's relevance, but developing that into a report suitable for a client and court is another matter.
I still bow to more experienced examiners regarding this - a colleague recently reviewed one of my reports, and changed my use of the past tense to present through out … His logic, which I have to agree with, is that the data is still that way, although it is no longer in the computer, it still exists in the present …
Even something as simple as an agreed list of terminology and explanations for a glossary would be a good start.
I'm starting to ramble a bit, but I'm glad that the discussion is starting again, and I hope that it will progress further than just talk.
Happy St.Georges Day 😉
Simon