data collection, Proof of Concept, human research protection, individuals, data analysis, Evaluation Working Group, collection, mobile technologies, ethical requirements, research project, individual, Testing & Evaluation Working Group, Evaluation team, MoLE Project, Mobile Learning Environment, mobile learning, interpretation, Human Research Protection Approach, requirements, European Union, Human Research, US Department of Defense's Coalition Warfare Program, Protection of Human Subjects, informed consent, ethical principles, institutional requirements, planned research, human participants, the MoLE, Testing and Evaluation, scientific research, Term of Reference, mobile content, Research Ethics, the Testing & Evaluation Working Group, The Testing, Medical Content, Testing, global capability, ethical misconduct, research paper, Human Research Protection Human subject research, Human Research Protection Program, human research subjects, humanities research, research methodologies, research development
Research Ethics in the MoLE m-Learning Program
Jacob Hodges Q&P Ltd London, UK [email protected]
Geoff Stead Tribal Group Cambridge, UK [email protected]
ABSTRACT This research paper discusses the Mobile Learning Environment (MoLE) Project, a unique and ambitious effort sponsored by the US Department of Defense
's Coalition Warfare Program (CWP) in partnership with over 20 nations. The mobile learning project explored the usefulness and effectiveness of using mobile technologies as a tool to support training activities in medical stability operations. The paper discusses the importance of global research ethics and social responsibility in the testing and evaluating of science and technology projects that span the globe. It provides an understanding of research ethics requirements and looks at the technical challenges in applying these requirements within a global framework. Finally, it showcases an integrated application of a mobile capability in accordance with a myriad of research ethics guidelines and concludes with the accomplishment of evaluating this global capability.
Author Keywords Defense, human research protection, research ethics, work-based learning, m-learning, training, smartphones INTRODUCTION Science and Technology (S&T) research has played a significant role in developing new technologies
that benefit both society and the defense sector. There are many positive impacts and the benefits have revolutionized our way of life. However, this has not always been the case, and there are countless examples of ethical misconduct in social and behavioral sciences and humanities research. Researchers in these genres often assert that regulations for the protection of human research subjects
do not apply. However, a close reading of the regulations regarding the involvement of human beings
will find references that state otherwise (32 CFR 219; SECNAVINST 3900.39D). ethical conduct
is an essential element in all scientific research
and is necessary to foster collaboration, cooperation and trust. It is imperative that research be socially responsible in order to make advancements in scientific knowledge
(Resnik, 2009). Research has been defined as a systematic investigation, which includes research development and testing and evaluation activities that are designed to develop or contribute to generalized knowledge (Title 45 CFR 46, Belmont Report). A human subject is defined as an individual who is or becomes a participant in the research, either as a recipient of the test article or as a control. With such a broad definition, researchers should ensure that if a research project involves any human being interactions, then moral and social dimensions should be considered. During the development stage, the project should incorporate a `gate-keeping' mechanism into the planning activities that demonstrates an endorsement of ethical practices, solid research methodologies and applicable professional standards. For cooperative research, the project's planning activities need to adhere to each of the institutional or country's research ethics requirements to ensure that the project addresses the moral and social dimensions (Swazey & Bird, 1997; 32 CFR 219). Therefore, when conducting research, three components are required in the research design
; specifically, the Human Research Protection Program, which ensures that the researchers promote the integrity of the research and safeguard against any misconduct; a data collection
Plan that will ensure that there is a clear understanding of the research objectives and develops trust in the data collection process; and the Data Analysis and Interpretation Process which builds ownership across the research project and provides safeguards against any misconduct or impropriety that might reflect on the researchers or organizations involved (Creswell, 2009). Human Research Protection Human subject research is research that involves a living individual about whom an investigator obtains data through interaction. This may include, but is not limited to, any type of communication, such as surveys, emails, internet, interviews, face-to-face, etc., between the individual and the researcher (SECNAVINST 3900.39D). Human research also includes risk management and the achievement of research objectives in areas relating to human safety, security, legal and regulatory compliance and governance (ANSI/ISO/ASQ Q10006-2003(E)). Human research protection includes a code of ethics to preserve individual autonomy, confidentiality, integrity, privacy, security, and respect while minimizing risks and discomforts. Any data collection procedure from an individual, directly or indirectly, should incorporate this ethical code within its informed consent document. This written document is viewed as a truthful and respectful conversation which outlines the research approach, and the rights and responsibilities of both the researcher and individual (Burgess, 2007).
Data Collection Ethical issues in data collection refer to the need to guard against the collection of harmful or identifiable information. To ensure unnecessary data is not collected, i.e., data that will not be used as part of the analysis or not required for research objectives, extensive collaboration is needed among the research team to ensure the data collection strategy is understood and accepted. A concerted effort is required to guarantee for all the prospective research participants from whom the data is being collected, that the information being collected is free from intrusions into their personal life, the data will not contain any identifiable information without consent, and that each person participating in the research has the right to `not answer' any questions without reproach. Surveys and questionnaires have their own ethical issues, since the collection process, especially using technologyenabled capabilities, has the potential to link identifiable information to the response. Researchers should be conscious of the potential for this to be intrusive and should seek to minimize any intrusion. The confidentiality of the data must be respected and protected by positive measures (Burgess, 2005). This requires the research team to identify any potential risk related to the privacy of the individuals and to convey this as one of the primary components in the informed consent. Therefore, from a data collection standpoint, the informed consent should identify (1) how the research protects the anonymity of the individuals; (2) the testing process, including roles and responsibilities; (3) the expectations of the individuals; (4) the evaluative process; (5) how data will be shared collectively to support other research initiatives, and (6) how ownership of the data collection process will ensure anonymity, privacy and confidentiality (Israel & Hay, 2006; Patton, 2002). Data Analysis and Interpretations At the outset, the link between the terms `ethical' and `statistical' are not self-evident. The Collins English Dictionary & Thesaurus defines statistics as a `numerical fact collected and classified systematically, and the science of classifying and interpreting information'. The definition of ethics is defined as the `conscience, moral values, principles, standards and rules of conduct'. Although there is not an apparent connection between statistics and ethics, the definition of ethics (i.e., rules of conduct) and statistics (i.e., fact collection and classification) is the key to the relationship. Ethical issues may arise in data analysis and interpretations. Researchers should show caution during data interpretation. Data analysis should be based on sound statistical research practices leading to conservative data interpretation which does not overreach or claim the data is more significant or important than they really are (Resnik, 2007). Even within this paradigm, the interpretation of the same data can take different pathways, none of which may be unethical. Differences in data interpretation may well benefit the scientific process and allow researchers to capitalize on the important debates that lead to new technologies. Highly collaborative projects, especially when collecting both qualitative and quantitative data
, must engage coresearchers during the data analysis, interpretation and report writing
process to preserve the integrity of the project's results and to ensure impartial interpretations of the data. This is particularly important in this project given the global nature and cultural diversity of the participants. To ensure reciprocity, all of the organizations involved should receive some benefit from the research as well as have an opportunity to provide input into the interpretation of the analysis. The research results should show openness, sensitivity, accuracy and objectivity in the choices of analysis and dissemination, to ensure it respects the interest of the different groups in society (Burgess, 2005). MOBILE LEARNING ENVIRONMENT (MOLE) PROEJCT The Mobile Learning Environment (MoLE) Project was a two-year Coalition Warfare Program (CWP). It was sponsored by the Commander, U.S. Naval Forces Europe (CNE)/Commander, Naval Forces Africa (CNA)/Commander, SIXTH Fleet (C6F) and co-sponsored by the Deputy Director, Joint Staff (J-7) for Joint and Coalition Warfighting (DD J7 JCW) Joint Knowledge On-Line (JKO) Director. The CNE-CNA-C6F Deputy Director for Training envisioned that a mobile learning capability could address the difficult challenges associated with training and communicating in the largest maritime area of operations (AOR) where there are also the challenges of low-bandwidth, limited internet connectivity and limited infrastructure. The DD J7 JCW JKO Director viewed mobile technologies as a critical step in meeting his organization's requirement to facilitate and provide training to the U.S. and its multinational partners. The basic concept was that the MoLE Project would leverage the global cellular network infrastructure, mobile technologies and emerging mobile applications/service models to build a mobile learning (m-Learning) capability that integrated into the DD J7 JKO portal. It would provide the foundation for conducting a proof of concept for evaluating a mobile learning (m-Learning) solution for meeting emerging training requirements that not only exists in the sponsoring organizations, but many related departments, initiatives and partnerships. Through the proof of concept it would demonstrate an enhanced interoperability and yield high benefits to all the global partners involved by providing general and medical education and training to military and related civilian personnel of countries in need of humanitarian and civil assistance, joint exercises and force training, or other types of on-demand training. This would, in turn, be shared by the international community
to support their medical education and training as well as initiate the development of a sustained capability within their own country's defense learning organizations. 2
In order to support the MoLE Project goals and objectives, a Testing and Evaluation Working Group was established, which consisted of representatives from each of the 22 nations. The working group was divided into three teams to address the challenges related to research ethics, specifically: human research protection, data collection, and data analysis and interpretation. Human Research Protection Approach As a cooperative research project, MoLE faced several challenges since it involved 22 institutional requirements as well as county-specific guidelines. During the planning phase, a human research protection team was established to ensure the research ethical requirements of each institution was identified and subject matter
experts were involved to ensure a solid understanding of applicable directives. There were several key US Directives related to the Protection of Human Subjects and Adherence to Ethical Standards in DoD-Supported Research. Collectively, these guidelines use the ethical principles outlined in the Belmont Report for the foundation for protecting individuals involved in research, which includes respect for persons, education and training, informed consent, vulnerability of individuals, collaborative research, etc. However, since MoLE was a research project conducted in an established educational setting that involved "research conducted on the effectiveness of or the comparison among instructional techniques, curricula or classroom management methods," it was exempt from a rigorous review (32 CFR 219.101(b)(1)). A review of the UK requirements stated that research involving human participants undertaken, funded or sponsored by the Ministry of Defence (MOD), must meet acceptable ethical standards and that these ethical standards are upheld by the MOD Research Ethics Committees (MODREC). Their Joint Services Publication (JSP) sets out in the MOD instructions requirements for the ethical conduct and treatment of human participants in MOD research (both clinical and nonclinical) and the ethical treatment of human participants. The JSP states that the directive applies to the conduct of research to collect data on an identifiable individual's behaviour, either directly or indirectly (such as by questionnaire or observation); this makes it applicable to the MoLE Project. In ensuring that the project met the requirements required by the European Union (EU), several documents were used as reference, specifically: the European Union (EU) Data Protection Requirements and the Privacy and Human Rights; and Comparing the United States to Europe and EU Data Protection Directive 95/46/ED. The basic Ground rules
for privacy, according to both documents state that all individuals involved in a research project need to be informed about the planned research use of collected data, independently of the type of data collected. If a survey is planned within the project, individuals need not only to be informed of how their data is planned to be handled but also provide appropriate authorization. In addition, the survey design must guarantee that only data specifically required for the purpose of the research project will be gathered, unless clearly stated otherwise. After careful consideration of all the aforementioned documents and directives, and email exchanges amongst the international participants, it was determined that the most restrictive guidance was from the EU; therefore, an informed consent would be required that incorporated both US/EU ethical requirements. The MoLE Informed Consent was developed to include all research protocol areas (i.e., Introduction, Purpose of the MoLE Project, Duration of Participant Involvement, Procedures, Testing and Evaluation Process, Risk and Discomforts, Potential Benefits, Voluntary Participation and Withdrawal, Confidentiality, and Consent of the MoLE Individual). Data Collection, Analysis and Interpretation Approach At the project kick-off meeting, the Testing and Evaluation Working Group held an intense meeting in order to develop a stage-gate approach for ensuring ethical practices were utilized throughout the data collection, analysis and interpretation. The participants were separated into the three groups so that equitable efforts would be placed on the data collection, analysis and interpretation requirements. However, since this was the first meeting, the focus was changed to concentrate more on `what should be asked' and `what data should be collected' in order to achieve success. As a result of the collaboration, the team decided to put forth a concerted effort on (1) what types of questions should be asked; (2) the testing and evaluation process, including roles and responsibilities; (3) research expectations; (4) the evaluation process; (5) how the data will be shared during the analysis phase; and (6) how transparent data collection would ensure anonymity, privacy and confidentiality. RESEARCH STRATEGY After the project kick-off meeting, the Testing and Evaluation Working Group focused on addressing issues relevant to determining how the MoLE Project would measure if it had met its goals and objectives. The working group was essentially divided into three teams to ensure the key elements in evaluating the Proof of Concept were identified and resolved (e.g., human research protection, data collection, data analysis and interpretation). The three teams independently collaborated via quarterly teleconferences and net meetings, with the exception of the human research protection team who collaborated on a weekly basis to ensure all ethical requirements were identified and documented. 3
Human Research Protection Subsequent to the review of pertinent documents ranging from `the involvement of individuals in research' to `information and data collection requirements', the human research protection team developed a draft MoLE Proof of Concept (PoC) Informed Consent which would serve as the written agreement between the researchers and the individuals. It was then emailed to all Testing & Evaluation Working Group members for review and comment in order to ensure complete understanding of the research protocol. Comments were then incorporated into the document and presented at a three-day Testing and Evaluation Working Group meeting to ensure the document reflected the `as-is' testing and evaluation process. This review process ensured that the research protocol was totally accurate, including how testing and evaluation would be carried out, duration of involvement and confidentiality, etc. The MoLE PoC Informed Consent Form was subsequently resent via email to over 40 international delegates for their final review and feedback. The email also requested each delegate to consider if a version of the informed consent should be translated into their native language to ensure complete understanding of individual requirements and expectations. The informed consent was thus made available in English and also translated into Spanish and French.
Data Collection, Analysis and Interpretations During the project kick-off meeting, the Testing and Evaluation team members
that were not on the human research protection team formed small groups to collaborate on what type of data should be collected in meeting project goals and objectives. Each of the groups provided recommendations on the types of data that could be collected and operational measures for determining effectiveness and performance. However, since the mobile content was still being developed it was not yet possible to finalize this. In focusing on the core goals and objectives of the MoLE Project, the two teams determined that data collection should focus on four themes, as shown in Table 1. An email was sent to each of the Testing & Evaluation team members asking them to provide five questions they would ask if they were developing the survey. As a result, an average of 150 inputs per Term of Reference in addition to over 20 other potential questions, were identified.
Term of Reference Accessibility Self-Efficacy Usefulness Utility
Meaning The degree to which a mobile training applicable is available to the user A user's belief that he/she is capable of producing the desired outcome of the task required The benefit or availability of mobile technologies in providing training The effectiveness, or practicality, of using mobile technologies in providing training
Table 1 - MoLE Terms of Reference
Once the Medical Content Working Group identified the medical content, a three-day meeting of the Testing & Evaluation Working Group was convened, which included the Medical Content Working Group Leads. The purpose of this meeting was to (1) inform the Testing & Evaluation Working Group of the medical content decision; (2) identify the scenarios and vignettes to ensure the Proof of Concept included both medical and non-medical participants; (3) identify the Survey questions
that would be asked as part of the scenarios and vignettes; (4) collaborate on the transparent data collection strategy; (5) review the MoLE PoC Informed Consent and how it fits into the data collection strategy; and (6) to develop the MoLE Testing & Evaluation Plan.
After the first day the group was divided into two groups, specifically: the data collection team, and the data analysis and interpretation team. The data collection team, in collaboration with the Medical Content Leads, developed a storyboard of how the data collection (e.g., survey and transparent digital data) would integrate with the medical content. In the meantime, the data analysis and interpretation team focused on the how the data would be statistically displayed as well as how the data could be made available to international researchers for follow-on research initiatives. At the end, each of the teams, including the human research protection team, briefed the content and technical teams on their results to enable integration into the mobile apps under development.
At the start of the second year, the Testing & Evaluation Working Group convened for a final meeting before the completion of the technical development to finalize the data collection, analysis and interpretation strategy. Discussion focused on refining the process of how the Proof of Concept would function. Final decisions were made regarding the demographic data collection plan, how individuals would participate in the Proof of Concept and the human research protection issues.
APPLICATION DEVELOPMENT Tribal Group was responsible for developing the mobile application based on the requirements from each of the three Working Groups
, specifically: learning content
, Technology & Transition, and Testing & Evaluation. The learning content team worked with a wide range of medical and training stakeholders to design, convert, import and create mobile content to support the needs of the target users. The Technology & Transition team developed a cross-platform toolset that enabled mobile learning content to be deployed to apps on both Android and iOS (apple) platforms, and worked with JKO (the US DoD e-learning platform) to integrate the mobile platform with their back-end infrastructure. The Testing &
Evaluation team ensured that the testing and evaluation process was carried out as planned, specifically regarding the ethical research issues and human research protection issues. EVALUATION PROCESS Participation in the MoLE Proof of Concept trialing was a two-step approach. First, each organization participating in the project was required to identify at least 20 individuals to participate in the trial. Email addresses of each volunteer would be sent to the MoLE Research Ethics Coordinator who was the only person with full access to all volunteer contact details. Using a tool developed by Tribal, the Research Ethics Coordinator would generate a unique pin number for each volunteer, and email each one with a welcome email containing all instructions for participation. This included a link to their registration page, their personal identification number (PIN), and additional links to support their involvement (i.e., introduction video, user guide and detailed overview and installation guide). The email provided a link to the MoLE Registration Site. In the Registration Site, no data could link the PIN number with the email or name of the individual. On two or three occasions, volunteers sought support from the MoLE Technical Help Desk and inadvertently included their PIN number. In such cases, they were reassigned another PIN number to ensure their anonymity. Once the app was installed they could participate in the MoLE Proof of Concept trial. When the MoLE Proof of Concept trial had ended, the Research Ethics Coordinator deleted all databases that contained any references to email addresses and PINs. Figure 1 - MoLE Proof of Concept Process As Figure 1 shows, the testing and evaluation process was broken down into six steps. First, an email announcement would be provided to each individual. The individual would register online, install the app and then activate their app using their personal identification number (PIN). Each individual was afforded the opportunity to become familiar with the app before starting the evaluation. During the online registration each individual was required to acknowledge the MoLE PoC Informed Consent and complete the demographics questionnaire (see Figure 2). This was a required step. Without acknowledgement they were unable to validate their PIN number or activate the mobile app. 5
Figure 2 - Demographics Data On completion, the PIN number would be activated, and the user would be able to download the Global MedAid app from their local App Store, and register using their unique PIN. This PIN only carried a national identifier (to permit localization), and no personally identifiable data. Once the Global MedAid app was installed, volunteers were encouraged to familiarize themselves with the app, and explore the various features. When they were ready to begin the evaluation, they launched a specially designed "evaluation layer", which set them specific tasks within the app, following pre-defined medical scenarios / vignettes. Their use of the app was monitored (transparent data), and their feedback on the task collected (via an in-app survey). This data was then synchronized back to the project website, and collated. Figure 3 shows a sample page from the evaluation survey. The volunteer is asked to complete a task and on completion answers a few questions. Most of the questions were completed with a "slider", allowing selection against a 7-point likert scale. All questions offered the option of not responding and required an active effort to select an answer. Figure 3 - MoLE PoC Response Format Volunteers were asked to complete three vignettes/scenarios representing three different ways that the Global MedAid app might be used: one before deploying on a humanitarian mission, one en-route, and one on arrival. Each scenario was structured in the same way, setting the user a series of tasks, tracking their activity, and recording their answers to specific questions. At the end, they were asked one final set of questions requiring text input, and offering a more open format for responding. At a pre-defined date the PoC was completed and no more tracking data or Survey Responses
were collected. A new version of the app was subsequently released that automatically upgraded, removing the evaluation survey and registration requirement of the app. Once the Proof of Concept evaluation was completed, the Testing & Evaluation Working Group initiated an analysis of the data in which several group members independently conducted their own analysis of the data. In addition to the internal analysis that was being conducted, several outside resources were used to provide their own independent analysis of the interpretation of the data. In areas where there were disagreements, additional analysis was conducted to help identify the potential disagreements in the analysis and draw appropriate conclusions. The results of this evaluation will be reported on in a separate report. This paper aims to describe the process and clarify the steps taken to insure clearly defined research ethics, and protection of volunteer data. VOLUNTEER NUMBERS At the conclusion of the evaluation period, 268 individuals had registered, then downloaded and installed the app to participate; 70.9% used their own mobile device whilst 29.1% borrowed a colleague's device. Overall, 177 (66.4%) individuals actually started the evaluation and 137 (51.12%) completed it, from 21 countries. A majority (63%) were using an iPhone 4 or 5, and 37% were using Android (2.2, 2.3, 3 or 4). 6
The individuals' professional expertise, based on the demographics survey, showed that 34.2% had medical experience, 26% were involved in e-Learning, 25.3% identified their professional expertise as `other', 11.2% conducted general training, 1.1% were involved in rescue effort, 2.2% were involved in rescue operations and 3% declined to answer the question. A majority of the individuals were using the app in English; German, French, Georgian and Spanish versions were also available. THE WIDER PROJECT The primary purpose of this project was to explore the utility and effectiveness of using mobile technologies and then to create a transition strategy that moves mobile learning and training into the mainstream of defense training for all of the international partners involved in the project. This process has already started and several partners are adopting some of the system and processes that will include some of the content created by the project into their mobile training programs. Although the project officially ended on 30th of September 2012, the final report will not be released until the international partners have an opportunity to review the Proof of Concept findings so that they have ownership of the project and results of the analysis. CONCLUSIONS The Mobile Learning Environment (MoLE) Project, as a global research initiative, demonstrated that project goals and objectives can be fulfilled by employing ethical and social responsible practices. Given the diverse groups of stakeholders, there was considerable complexity to develop and implement a test and evaluation strategy that incorporated research ethics guidelines. Although there were many challenges in integrating ethical research requirements into the technical interfaces, with careful management and an effective stage-gate approach these challenges can be dealt with clearly and effectively. ACKNOWLEDGMENTS The authors would like to acknowledge the subject matter advice on EU/UK Research
Ethics by Dr. Andrey Girenko (Deutsche Forschungszentrum fьr Kьnstliche Intelligenz GmbH (DFKI) Center for e-Learning Technology (CelTech)), Dr. Andrea Loesch (GIRAF PM Services GmbH) and Dr. Venkat Sastry (Defence Academy of the United Kingdom (DAUK), Dr. Tammy Savoie (J-4, USAF), the international partners and the MoLE Testing & Evaluation Working Group Members. This work is related to a Department of the Navy (DoN) Grant (N62909-11-01-7025) and two DoN Contracts (N6818111-P-9000 and N68171-12-P-9000) which were awarded by the Office of Naval Research Global (ONRG) and funded by the Coalition Warfare Program. The views expressed herein are solely those of the authors and do not represent or reflect the views of any Government Agency
or organization that participated in the project or mentioned herein. References Anderson
, B. (2004). Bringing Business Ethics to Life: Achieving Corporate Social Responsibility
. ASQ Quality Press ANSI/ASQ Z1.13-1999: "Quality Guidelines for Research". http://standardsgroup.asq.org ANSI/ISO/ASQ Q10006-2003(E), "Quality management systems Guidelines for quality management in projects". May 12, 2006. Belmont Report: Ethical Principles and Guidelines for the Protection of Human Subjects in Research. The National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. DHEW Publication No. (OS) 78-0012. Bernhart, M.S., and Maher, F.J. (2011). ISO 26000 in Practice: A Users Guide. ASQ Quality Press Burgess, R. G. (2007). The Ethics of Educational Research. The Social Research and Education Studies Series: 8. Routledge Falmer Collins English Dictionary & Thesaurus (2006). 2nd Edition. HarperCollins Publishers Common Ground: Quality and Social Responsibility. (2007). American Society for Quality (ASQ) White Paper
Directive 95/46/EC of the European Parliament and of the Council of 24 October 1995 Directive 2002/58/EC of 12 July 2002 Department of Defence (DOD) Directive 3216.2: "Protection of Human Subjects and Adherences to Ethical Standards in DoD-Supported Research". January 7, 1993 7
Department of Defense (DOD) Instruction 3210.7: "research integrity
and Misconduct". May 14, 2004 Department of Defense (DoD) Instruction 5400.11: "DoD Privacy Program". May 8, 2007 Dumortier, J. and Goemans, C. (2002). Roadmap for Advanced Research in Privacy and Identify Management (RAPID). Interdisciplinary Centre for law and ICT (ICRI) Ethical Review in SEVENTH Framework (FP7) European Convention for the Protection of Human Rights and Fundamental Freedoms. Council of Europe
, Rome 1950. http://conventions.coe.int/Treaty/EN/CadreListeTriates.htm Federal Policy for the Protection of Human Subjects, 45 CDR Part 46 (Common Rule). http://www.hhs.ohrp/humansubjects/commonrule/index.html Federal Policy for the Protection of Human Subjects, 32 CFR 219: Protection of Human Subjects. http://www.gpo.gov/fdsys/pkg/CFR-2007-title32-vol1/content-detail.html Federman, D. D., Hanna. K.E., and Rodriguez, L.L. (2003). Responsible Research: A Systems Approach to Protecting Research Participants by Committee on Assessing the System for Protecting Human Research Participants. The National Academy Press Government-Wide and DOD Definitions of R&D. (2012). http://mattcegelske.com/u-s-government-wide-anddepartment-of-defense-dod-definitions-of-research-and-development-rd-acquisition/ . Hesse-Bieber, S. N. and Leavy, P. (2006). The Practice of qualitative research
. Thousands Oaks, CA. Integrity in Scientific Research: Creating an Environment That Promotes Responsible Conduct. Committee on Assessing Integrity in Research Environments, National Research Council
, and Institute of Medicine ISO 31000, "Quality management systems Guidelines for quality management in projects". May 12, 2006. https://standardsgroup.asq.org Israel, M. and Hay, I. (2006). Research ethics for social scientist: Between ethical conduct and regulatory compliance. London: Sage Office of Naval Research (ONRG) Historical Timeline. http://www.onr.navy.mil/en/About-ONR/History-ONRTimeline/nobel-laureates.aspx. 31 Aug 2012. Overview of national legislation
in over 50 countries: An International Survey of Privacy Laws and Developments. Electronic Privacy Information Center and Privacy International. http://www.privacyinternational.org/survey Patton. M.Q. (2002). Qualitative evaluation and Research methods
. 2nd Edition. Newbury Park, CA: Sage Research, Development, Test and Evaluation Appropriations. Department of Defense (DoD) Financial Management Regulations. Volume 2B. Chapter 5 Resnik, D. B. (1998). The Ethics of Science. Routledge. New York. SECNAVINST 3900.39D: "Human Research Protection Program". 6 November 2006 Shamoo, A.E., Resnik, D. B. (2009). Responsible Conduct of Research, 2nd Edition. Oxford University Press. Swazey, J.P., and Bird, S.J. (1997) Teaching and Learning Research Ethics, in Elliott, D. and Stern, J.E. (eds.) Research Ethics: A Reader, University Press of New England
, Hanover Westcott, R.T. (2004). Simplified Project Management
for quality professionals: managing small and medium-sized project. ASQ Press 8