Documentation and Testing Meeting 2010-08-11

Notes from Documentation and Testing Workgroup
Date: August 11, 2010
Time: 2:00pm-3:00pm
Attendees:
Janet Campbell, Arien Malec, Tony Calice, Keith Boone, Dragon Bashyam, Parag More More, Douglas Pratt, Karen Witting, Will Ross, Uvinie Hettiaratchy

Actions

#	Date	Action	Status	Owner	Due Date
11	08/11/10	Form plans to work with other workgroups for last ½ hour of “-a-thons” time on Tuesday morning of Face to Face, will touch base with Brian Behlendorf	Open	Janet	8/16/10
12	8/11/10	Will draft higher level policy guide and put out a call to people who want to get involved	Open	Janet	8/16/10
13	8/11/10	Write abstract for XDD developers’ guide	Open	Janet	8/16/10
14	8/11/10	Develop a list of documents the WG wants printed in advance of next week’s Face to Face meeting	Open	Janet	8/16/10
15	08/11/10	Will own Content Security for Simple Heath Transport. Will write abstract and start a wiki page.	Open	Arien	8/16/10
16	8/11/10	Will use the wiki page to start reviewing others’ work and offer comments and will use the mailing list if they want their documents reviewed.	Open	WG	8/16/10

Agenda
 
 

1. Review Documentation Priorities survey results
   1. Make sure everyone's on board with the artifacts that were voted N/A
   2. Review and confirm that Priority 1 artifacts are still Priority 1
   3. Assign volunteers for Priority 2 and Priority 3.
2. Check in on progress for all documents currently under editing
3. Plan for Docuthon at Face to Face

Notes

Janet Campbell
· Last week’s major goal was prioritizing all the documentation on the table.
· Mostly distribution of priorities was vague, with opinions split between priorities 2 and 3, so after some discussion we can shift things down.
· 3 documents raised a lot of questions: why are we doing this? What exactly is this?
§ For this meeting she wants to review why/if they are priorities.
· First document: SMTP Clients Development Guide, which includes use of reference implementation libraries.
§ Interesting because large range of opinions about this than the other two documents.
§ Could someone describe?
Arien Malec
· Sees this doc as a more general item depending on how the client libraries come out. This is more than the API. It gives a reference guide, a how-to guide, how to install Java and complete basic tasks. If you are an EHR developer and want to take and download a library and put in source tree using API, this is the doc.
Janet Campbell
· So the client code needs to be well established before this can be written?
Arien Malec
· Yes, sees this as a third priority, first needing the code.
Janet Campbell
· Next is the XDD Priority guide.
o Overall priority #2 by consensus.
Arien Malec
· Fair statement. They are not yet ready to write the document.
Karen Witting
· Objects. Feels we don’t have an understanding of where the code is.
· Sees the XDD document having two purposes:
o One, collect everything that is already written, and consolidate in one place.
o Two, find the need that the code would fill. Doesn’t understand why we need to wait for coders to act, unless it is decided that they will be making decisions instead of the documentation and testing or other groups.
Arien Malec
· XDD specialization is separate, and is at priority 2.
Karen Witting
· Retracts objection.
Janet Campbell
· But it’s true that the people writing clearly and the people figuring out the code need to come together to make sure ends meet.
· Last document marked inapplicable was the Testing Guide.
Arien Malec
· Introduces certification/performance Guide.
· Should we retire the testing guide in favor of the conformance Guide?
Keith Boone
· Conformance guide: tells you what to do.
· Testing guide: tests if something does what it says it will do. More about test scripts, documentation about test units.
· So in this context, conformance guide is meaningful use guide and test guide is test scripts.
Janet Campbell
· Are we also writing unit scripts?
Arien Malec
· Yes, Ref Imp should write unit scripts.
· Testing guide is especially about end-to-end testing.
Parag More
· Sees the line as getting blurry, as John had commented. Would like to know more about the distinction.
Janet Campbell
· Can we do that now?
Parag More
· Thinks the conformance guide is more at the behavior level and these are the basic set of paths it has to perform. John pointed out that based on the deployment model, there should be a different level that should be reached in the guide.
Keith Boone
· Deployment models=typologies.
· Testing for interactions.
Janet Campbell
· When describing results of tests, is that in conformance guide or testing guide?
Keith Boone
· Conformance guide says which behaviors/functions need to be exhibited.
· Testing guide tells how to conduct a test if you want to test something.
Janet Campbell
· So, in other words, when sending a message from first to second point, this shall do this, this shall do that.
· In order to test, I could send a message from here to here, and it should be able to do x behavior.
Keith Boone
· Yes, you could have conformance for something that can’t be tested.
Tony Calice
· Thinks it could be valuable if we could develop the distinction with observable results. Wants it to provide guidance on appropriate use of HISPs.
· The test guide would show how well is it conforming to our objectives. Do we need the detailed steps or just the functional level things that we would be able to do with it?
Arien Malec
· Yes, much more integrated and detailed steps not needed.
Janet Campbell
· By giving these inputs, one is proving the behavior.
· Seems testing guide grows out of the conformance guide.
Arien Malec
· Yes, and both need to work hand in hand with the Reference Implementation WG.
o Make sure the Java code and the CSharp code are functionally identical.
o As we proceed should work in conjunction with Ref Imp WG to make sure we have sufficient proof the two functions do the same thing, that they do work end to end.
Keith Boone
· Differs. Specification should influence the implementations.
· Final result should include interoperability testing process, do the testing, verify that things follow the steps and perform appropriate steps for implementations, or else could wind up with tail wagging the dog.
Arien Malec
· Doesn’t disagree but sees working together as a useful way of constraining reference implementations.
· Useful to check behavior and mutually check on both sides.
· Agrees that specifications should drive implementations.
Janet Campbell
· But need to make sure we are still working with them so they know what they are being driven by.
· Who in this group is also on Ref Imp WG?
Keith Boone
· John is on all call. We (GE) have basically provided the code for some of the transformations and our developers have been assigned to another task, so we are no longer doing reference implementation work, but if you’d like to have us more engaged to provide a check on things are being implemented, we could find a way to do that.
Janet Campbell
· We all could get involved in the writing of the specs (what these things should do) depending on how face to face goes.
Keith Boone
· And by using the NHIN Direct wiki.
Parag More
· Vince Lewis from his organization has been contributing. They touch base offline and Vince has been looking at the conformance guide. But Vince isn’t using it as a guideline.
Janet Campbell
· Might not hurt if on Tuesday morning at the Face to Face they check in with other workgroups during last ½ hour. Will make plans for more crossover.
Arien Malec
· Wants to be on top of the issue Keith raised, the tail wagging the dog.
· Umesh and Sean wrote one of the implementations and first draft of specs, so there is a danger of not enough involvement from the documentation people.
· Need the process to be solid.
Janet Campbell

• àWill touch base with Brian Behlendorf.

Round: clarifying the difference b/w conformance guide and testing guide

Tony Calice	It appears both are important and something should be in place to have the conformance guide as a near-term deliverable, echoing Keith.
Dragon	No comment.
Parag More	Distinction between the two--what do we have in the conformance guide? Not sure if anyone has had a chance to look at it. Is that the level we should continue at? Not in terms of abstract model? Not clear on the language and the way we want to frame the guide. He understands testing guide is more about end-to-end interoperability works, but is still struggling with how to frame that conversation. Arien Malec · If I understand Keith correctly, conformance guide is defining on top of the specifications, conformance statements for what an implementation must do. For instance, language describing the “musts and shalls” for including an XDD file. It is used when there are specific components providing additional specifications about how to use. · Testing guide provides set of end-to-end tests on much more of a script level. For example, if you start with internet message format with xyz components with abc trust circles, etc…you should expect to see X at the other end of the system. It is a set of test situations and test scripts that enable confirmation that a system is interoperable. Parag More · That is what the conformance guide is talking about. John was challenging that. Is there an alternate framework we should look at for the conformance guide? There is a value 1 deliverable…so what are the next steps? Keith Boone · Believes John was referring to a ref imp of security agent as opposed to more focus on conformance guide of specific interactions. John is changing what he has proposed and is in the process of making changes that will address Parag More’s question. Parag More · Will wait for changes.
Doug Pratt	Are we looking at different levels of conformance? Specification—so conformance guide defines different levels and provides testable statements. o If no, is a spec not enough? As in do I conform to specification or not? o If yes, we would want to make sure anything in there is testable criteria. Keith Boone · We are using unconditionally compliant and conditionally compliant = shall/shall not · Same language used in all other circles Doug Pratt · Do we need separate specifications for conformance guide? Keith Boone · APA, SMTP, client developer’s guide…doesn’t know what Doug means. Doug Pratt · What is going on the wire, which calls are made: how do I craft a conformance statement around what is here? Arien Malec · Seems like Doug is describing the testing guide. If you do X action, Y will happen. Gives you the test set-ups (pre conditions), the test scenario and expected output. Doug Pratt · So performing such action shows test is passed? · Where is a statement crafting the body of the message? Keith Boone · Those are two different things: implementation guide shows how you craft it. Conformance guide says what the rules are. Rules don’t tell you how to do something. They are just the rules and have to be followed, without explanation. Implementation guide is really more about how/why to do something.
Karen Witting	Nothing to add.
Will Ross	Thought it was a lot clearer when conformance guide was defined as chunky rather than granular.

Janet Campbell
· Thinks explanations make sense. If we want to keep testing guides as priority 2, we probably need to be doing that relatively late. Doesn’t know if the conformance guide is in a place where we could start the testing guide.
· Does anyone disagree? Anyone want to own the testing guide?
Arien Malec
· Cannot declare the implementation done until we’ve got the conformance and testing guides done.
Janet Campbell
· We should focus on the conformance and testing guides during the –a-thons.
Keith Boone

• Will provide additional assistance can't take responsibility for whole thing.

Janet Campbell
· Other stuff: priority two documents that don’t yet have owners.
· XDD developers guide moves down to priority three, leaving XDD implementation as priority two.
· Also, SMTP and script implementation guides.
Arien Malec
· People in this group need to be working closely with implementation geographies.
Janet Campbell
· Email client considerations guide?
Arien Malec
· Thinks fits with first category of guest imp/client imp.
· Thinks Will would be a good person to own and bridge the gap between Documentation and Testing WG and Implementation Geographies WG.
Will Ross

• What about John Moehrke?

· Three folks this morning at the Implementation Geographies WG meeting said they would go between the different a-thons, but cannot remember who.
Janet Campbell
· Still priority two, not priority one.
· Maybe putting out an outline or structure.
Arien Malec
· Will write volunteer abstract, outline and start wiki page.
Janet Campbell
· Policy questions
· Janet Campbell will write higher level policy guide and will put out a call to people who want to get involved.
· Now have owners on the “2s”
· Can write abstract for FPD/XDD developer’s guide
Arien Malec
· Question: we have two more people helping out from a staff perspective.
à Wants a list of docs we want pre-printed for view during document-a-thon.
· Going through the docs, annotating, looking at physical copies as a great process to getting to final documents you think are close to deliverable
Janet Campbell
· NHIN Direct Overview, Security, and Conformance Guide

• Content Security for Simple Heath Transport.

· Question: owner for previous doc?
Arien Malec
· Will be owner. Umesh and Sean wrote the first draft.
Janet Campbell
· Still call for as many edits done prior to the process as possible.
· Anyone want a document edited?
(No one)
· So this means people feel the docs are fine, at the level they need to be at for Aug 17
·Encourage folks to go to the wiki page, start reviewing, offer comments.
· Use mailing list if you want people to review docs. There are people who want to help but can’t do full-time.
· Agenda for document-a-thon
o One document at a time
o Conformance guide
o Testing guide
o Communicating and representing to the ref imp group