Testing Guides

State: Draft

Introduction

This Testing Guide provides a plan for how one would test that their implementation is compliant with the Direct Project Specification. This Testing Plan consists of validating that your implementation can interoperate with at least three other implementations of various Deployment Models. The goal is to prove heuristically compliance with the Direct Project Specification.
 

Testing Overview

The system being tested is referred to as the "System Under Test" (SUT). It is expected that this is a complete system capable of Sending and Receiving using the Direct Project Specification. We use 'system' as the deployment models show any Sending side or any Receiving side is often made up of multiple distinct components. It is the whole set of components working together from end-user experience to the point at which the Direct Project Specifications that is collectively referred to as the System Under Test (SUT).
 
  This Test Guide instructs the tester to test the SUT against at least three other implementations of a different type. These other implementations will be referred to as the "Other Test System" (OTS). By testing against three other systems we get a good heuristic to show that the Direct Project Specification was implemented correctly.
 
 

Test Plan

Pre-Conditions: Prior to testing the following must be available.

• Endpoint Addresses must be known and recorded
• Endpoint Digital Certificates must be issued
• Endpoint Digital Certificates must be published in a way that the SUT can discover (sometimes this is manual, LDAP, e-mail, or DNS)
• SUT must have basic network connectivity to the OTS (this can be over the internet, or in a lab)

Test Scenarios

Preconditions

Trust anchors: A0, B0,
Valid certificate paths:
A0->A1->A2
A2 and A1 have AIA extensions providing the chain, A1 is retrievable via HTTP, chain not precomputed at SUT
A0->A3
B0->B1

Invalid certificates:
E1 (has AIA but path not retrievable)
E2 (has AIA, path retrievable)
A0->E3 (expired, via CRL)
A0->E4 (expired, via OCSP)

Certificate bindings (published in DNS or available in local trust store)
sunny.example.org, bound to A3
happy.example.org, bound to A2
bob@valley.example.org, bound to B1

Positive Tests

Test	Scenario	Expected behavior
Same address Organizational certificate	[[1]] receives message from [[2]]	Success, delivery, MDN
Same certificate path		[[3]] receives valid signed and encrypted message from [[4]]	Success, delivery, MDN
Different mutually trusted certificate paths Address bound certificate	[[5]] receives valid signed and encrypted message from [[6]]	Success, delivery, MDN
AIA certificate path building	[[7]] receives valid signed and encrypted message from user_[[8]]	Success, delivery, MDN

Negative Receipt Tests

Please note that many of these tests will intentionally require the system to receive a message that should be impossible to generate by the same system. Nothing in these these tests should imply that the SUT should create the negative test messages.

Test	Scenario	Expected behavior
Unencrypted payload	[[9]] receives unencrypted message	No delivery, no MDN, Audit, error retrievable at SUT
Encrypted, no signature	[[10]] receives encrypted message without a signature	No delivery, no MDN, Audit, error retrievable at SUT
Encrypted, bad signature	[[11]] receives encrypted message with corrupt signature	No delivery, no MDN, Audit, error retrievable at SUT
Message not verified	[[12]] receives encrypted message where content does not match signature	No delivery, no MDN, Audit, error retrievable at SUT
Corrupt encryption	[[13]] receives corrupted encrypted message	No delivery, no MDN, Audit, error retrievable at SUT
Chain can not be built	[[14]] receives valid signed and encrypted message signed by E1	No delivery, no MDN, Audit, error retrievable at SUT
Untrusted trust anchor	[[15]] receives valid signed and encrypted message signed by E2	No delivery, no MDN, Audit, error retrievable at SUT
Expired certificate	[[16]] receives valid signed and encrypted message signed by E3	No delivery, no MDN, Audit, error retrievable at SUT
Expired certificate	[[17]] receives valid signed and encrypted message signed by E4	No delivery, no MDN, Audit, error retrievable at SUT

Negative Send Tests

Test	Scenario	Expected behavior
Chain can not be built	[[18]] attempts to send to address bound to E1	No send, notification to sender, audit, error retrievable at SUT
Untrusted trust anchor	[[19]] attempts to send to address bound to E2	No send, notification to sender, audit, error retrievable at SUT
Expired certificate	[[20]] attempts to send to address bound to E3	No send, notification to sender, audit, error retrievable at SUT
Expired certificate	[[21]] r attempts to send to address bound to E4	No send, notification to sender, audit, error retrievable at SUT

Additional Negative Send Tests for Service Model

These tests deal with an organization operating as a Service Model Security User Agent.

Test	Scenario	Expected behavior
Unauthenticated Sender	Unauthenticated user attempts to send using SUT	No send, no notification to sender, audit, error retrievable at SUT
Receiver not managed by SUT	SUT receives encrypted payload for list of receivers none of whom are managed by SUT	No receipt, audit, error retrievable at SUT
SUT receives unencrypted payload	SUT receives unencrypted payload over an unencrypted channel (e.g., meant for other security user agents to send encrypted content)	No receipt, audit, error retrievable at SUT

Procedure

For each pairing of the SUT against each OS the overall procedure is:
 

1. Test the success scenarios where address, digital certificate, S/MIME, and content packaging are all functioning as expected
   1. send a document using the SUT to the OTS
   2. send a document using the OTS to the SUT
2. Test the failure scenarios to assure that they do not expose the document inappropriately and fail gracefully. Since some 'systems' will not allow the user to do some of these tests, it is acceptable to indicate that the system prevented the test
   1. If possible try to send a document from the SUT to the OTS without using the security provided.
   2. If possible try to send a document from a non-S/MIME e-mail client so that it is delivered as an attachment, but without security (S/MIME)
3. Test the options
   1. Configure the SUT to use TLS when connecting to the remote or internet mail server (this applies to both outbound SMTP as well as inbound SMTP/IMAP/POP as appropriate to the SUT deployment model).
   2. send a document using XDM zip file from the SUT to the OTS
   3. send a document using XDM zip file from the OTS to the SUT

Candidate Other Test system

The system under test will be tested against at least three of the following, preferably from different deployment models:
 

• Direct Project Java Reference Implementation
• Direct Project .NET Reference Implementation
• Off-the-shelf S/MIME proxy
• Eudora
• Microsoft Office Outlook
• Mozilla Thunderbird
• Any other e-mail system that supports S/MIME

Test Report

The Direct Project does not record success or failure of testing, but the test results should be documented and provided with any distribution or statements about the System Under Test (SUT).

• What Other Test Systems was it tested against?
• What modes of digital certificate distribution were tested?
• Was XDM option tested?
• Was TLS option tested?
• Document any test failures

---

1. Implementations need not sign and encrypt same-domain messages if they have local mechanisms for providing the same services that signing and encryption provides