Updated Guidelines for Evaluating Public Health Surveillance Systems

Guidelines Working Group CHAIRMAN
Robert R. German, M.P.H.
Epidemiology Program Office, CDC ADMINISTRATIVE SUPPORT
Dwight Westmoreland, M.P.A.
Epidemiology Program Office, CDC MEMBERS Greg Armstrong, M.D.
National Center for Infectious Diseases CDC Guthrie S. Birkhead, M.D., M.P.H.
Council of State and Territorial Epidemiologists
New York State Department of Health
Albany, New York John M. Horan, M.D., M.P.H.
National Center for Injury Prevention and Control, CDC Guillermo Herrera
National Immunization Program, CDC Lisa M. Lee, Ph.D.
National Center for HIV, STD and TB Prevention, CDC Robert L. Milstein, M.P.H.
National Center for Chronic Disease Prevention and Health Promotion, CDC Carol A. Pertowski, M.D.
National Center for Environmental Health CDC Michael N. Waller
National Center for Chronic Disease Prevention and Health Promotion, CDC The following CDC staff members prepared this report: Robert R. German, M.P.H.
Division of Public Health Surveillance and Informatics
Epidemiology Program Office Lisa M. Lee, Ph.D.
Division of HIV/AIDS Prevention -- Surveillance and Epidemiology
National Center for HIV, STD, and TB Prevention John M. Horan, M.D., M.P.H.
Office of the Director
National Center for Injury Prevention and Control Robert L. Milstein, M.P.H.
Office of the Director
National Center for Chronic Disease Prevention and Health Promotion Carol A. Pertowski, M.D.
Division of Environmental Hazards and Health Effects
National Center for Environmental Health Michael N. Waller
Division of Adult and Community Health
National Center for Chronic Disease Prevention and Health Promotion in collaboration with Guthrie S. Birkhead, M.D., M.P.H.
Council of State and Territorial Epidemiologists
New York State Department of Health
Albany, New York Additional CDC Contributors Office of the Director: Karen E. Harris, M.P.H.; Joseph A. Reid, Ph.D; Gladys H. Reynolds, Ph.D., M.S.; Dixie E. Snider, Jr., M.D., M.P.H. Agency for Toxic Substances and Disease Registry: Wendy E. Kaye, Ph.D.; Robert Spengler, Sc.D. Epidemiology Program Office: Vilma G. Carande-Kulis, Ph.D., M.S.; Andrew G. Dean, M.D., M.P.H.; Samuel L. Groseclose, D.V.M., M.P.H.; Robert A. Hahn, Ph.D., M.P.H.; Lori Hutwagner, M.S.; Denise Koo, M.D., M.P.H.; R. Gibson Parrish, M.D., M.P.H.; Catherine Schenck-Yglesias, M.H.S.; Daniel M. Sosin, M.D., M.P.H.; Donna F. Stroup, Ph.D., M.Sc.; Stephen B. Thacker, M.D., M.Sc.; G. David Williamson, Ph.D. National Center for Birth Defects and Developmental Disabilities: Joseph Mulnaire, M.D., M.S.P.H. National Center for Chronic Disease Prevention and Health Promotion: Terry F. Pechacek, Ph.D; Nancy Stroup, Ph.D. National Center for Environmental Health: Thomas H. Sinks, Ph.D. National Center for Health Statistics: Jennifer H. Madans, Ph.D. National Center for HIV, STD, and TB Prevention: James W. Buehler, M.D.; Meade Morgan, Ph.D. National Center for Infectious Diseases: Janet K. Nicholson, Ph.D; Jose G. Rigau-Perez, M.D., M.P.H. National Center for Injury Prevention and Control: Richard L. Ehrenberg, M.D. National Immunization Program: H. Gay Allen, M.S.P.H.; Roger H. Bernier, Ph.D; Nancy Koughan, D.O., M.P.H., M.H.A.; Sandra W. Roush, M.T., M.P.H. National Institute for Occupational Safety and Health: Rosemary Sokas, M.D., M.O.H. Public Health Practice Program Office: William A. Yasnoff, M.D., Ph.D. Consultants and Contributors Scientific Workgroup on Health-Related Quality of Life Surveillance
St. Louis University, St. Louis, Missouri Paul Etkind, Dr.P.H., Massachusetts Department of Public Health, Jamaica Plain, Massachusetts; Annie Fine, M.D., New York City Department of Health, New York City, New York; Julie A. Fletcher, D.V.M, M.P.H. candidate, Emory University, Atlanta, Georgia; Daniel J. Friedman, Ph.D., Massachusetts Department of Public Health, Boston, Massachusetts; Richard S. Hopkins, M.D., M.S.P.H., Florida Department of Health, Tallahassee, Florida; Steven C. MacDonald, Ph.D., M.P.H., Washington State Department of Health, Olympia, Washington; Elroy D. Mann, D.V.M., M.Sc., Health Canada, Ottawa, Canada; S. Potjaman, M.D., Government of Thailand, Bangkok, Thailand; Marcel E. Salive, M.D., M.P.H., National Institutes of Health, Bethesda, Maryland. Summary The purpose of evaluating public health surveillance systems is to ensure that problems of public health importance are being monitored efficiently and effectively. CDC's Guidelines for Evaluating Surveillance Systems are being updated to address the need for a) the integration of surveillance and health information systems, b) the establishment of data standards, c) the electronic exchange of health data, and d) changes in the objectives of public health surveillance to facilitate the response of public health to emerging health threats (e.g., new diseases). This report provides updated guidelines for evaluating surveillance systems based on CDC's Framework for Program Evaluation in Public Health, research and discussion of concerns related to public health surveillance systems, and comments received from the public health community. The guidelines in this report describe many tasks and related activities that can be applied to public health surveillance systems.

INTRODUCTION

In 1988, CDC published Guidelines for Evaluating Surveillance Systems (1) to promote the best use of public health resources through the development of efficient and effective public health surveillance systems. CDC's Guidelines for Evaluating Surveillance Systems are being updated to address the need for a) the integration of surveillance and health information systems, b) the establishment of data standards, c) the electronic exchange of health data, and d) changes in the objectives of public health surveillance to facilitate the response of public health to emerging health threats (e.g., new diseases). For example, CDC, with the collaboration of state and local health departments, is implementing the National Electronic Disease Surveillance System (NEDSS) to better manage and enhance the large number of current surveillance systems and allow the public health community to respond more quickly to public health threats (e.g., outbreaks of emerging infectious diseases and bioterrorism) (2). When NEDSS is completed, it will electronically integrate and link together several types of surveillance systems with the use of standard data formats; a communications infrastructure built on principles of public health informatics; and agreements on data access, sharing, and confidentiality. In addition, the Health Insurance Portability and Accountability Act of 1996 (HIPAA) mandates that the United States adopt national uniform standards for electronic transactions related to health insurance enrollment and eligibility, health-care encounters, and health insurance claims; for identifiers for health-care providers, payers and individuals, as well as code sets and classification systems used in these transactions; and for security of these transactions (3). The electronic exchange of health data inherently involves the protection of patient privacy. Based on CDC's Framework for Program Evaluation in Public Health (4), research and discussion of concerns related to public health surveillance systems, and comments received from the public health community, this report provides updated guidelines for evaluating public health surveillance systems.

BACKGROUND

guide immediate action for cases of public health importance;
measure the burden of a disease (or other health-related event), including changes in related factors, the identification of populations at high risk, and the identification of new or emerging health concerns;
monitor trends in the burden of a disease (or other health-related event), including the detection of epidemics (outbreaks) and pandemics;
guide the planning, implementation, and evaluation of programs to prevent and control disease, injury, or adverse exposure;
evaluate public policy;
detect changes in health practices and the effects of these changes;
prioritize the allocation of health resources;
describe the clinical course of disease; and
provide a basis for epidemiologic research.

Describe the public health importance of the health-related event under surveillance.
Describe the purpose and operation of the system.
Describe the resources used to operate the system.

indices of frequency (e.g., the total number of cases and/or deaths; incidence rates, prevalence, and/or mortality rates); and summary measures of population health status (e.g., quality-adjusted life years [QALYS]);
indices of severity (e.g., bed-disability days, case-fatality ratio, and hospitalization rates and/or disability rates);
disparities or inequities associated with the health-related event;
costs associated with the health-related event;
preventability (10);
potential clinical course in the absence of an intervention (e.g., vaccinations) (11,12); and
public interest.

List the purpose and objectives of the system.
Describe the planned uses of the data from the system.
Describe the health-related event under surveillance, including the case definition for each specific condition.
Cite any legal authority for the data collection.
Describe where in the organization(s) the system resides, including the context (e.g., the political, administrative, geographic, or social climate) in which the system evaluation will be done.
Describe the level of integration with other systems, if appropriate.
Draw a flow chart of the system.
Describe the components of the system. For example
--- What is the population under surveillance?
--- What is the period of time of the data collection?
--- What data are collected and how are they collected?
--- What are the reporting sources of data for the system?
--- How are the system's data managed (e.g., the transfer, entry, editing, storage, and back up of data)? Does the system comply with applicable standards for data formats and coding schemes? If not, why?
--- How are the system's data analyzed and disseminated?
--- What policies and procedures are in place to ensure patient privacy, data confidentiality, and system security? What is the policy and procedure for releasing data? Do these procedures comply with applicable federal and state statutes and regulations? If not, why?
--- Does the system comply with an applicable records management program? For example, are the system's records properly archived and/or disposed of?

Funding source(s): Specify the source of funding for the surveillance system. In the United States, public health surveillance often results from a collaboration among federal, state, and local governments.
Personnel requirements: Estimate the time it takes to operate the system, including the collection, editing, analysis, and dissemination of data (e.g., persontime expended per year of operation). These measures can be converted to dollar estimates by multiplying the persontime by appropriate salary and benefit costs.
Other resources: Determine the cost of other resources, including travel, training, supplies, computer and other equipment, and related services (e.g., mail, telephone, computer support, Internet connections, laboratory support, and hardware and software maintenance).

determining the specific purpose of the evaluation (e.g., a change in practice);
identifying stakeholders (Task A) who will receive the findings and recommen-dations of the evaluation (i.e., the intended users);
considering what will be done with the information generated from the evaluation (i.e., the intended uses);
specifying the questions that will be answered by the evaluation; and
determining standards for assessing the performance of the system.

Indicate the level of usefulness by describing the actions taken as a result of analysis and interpretation of the data from the public health surveillance system. Characterize the entities that have used the data to make decisions and take actions. List other anticipated uses of the data.
Describe each of the following system attributes:
--- Simplicity
--- Flexibility
--- Data quality
--- Acceptability
--- Sensitivity
--- Predictive value positive
--- Representativeness
--- Timeliness
--- Stability

detect diseases, injuries, or adverse or protective exposures of public importance in a timely way to permit accurate diagnosis or identification, prevention or treatment, and handling of contacts when appropriate?
provide estimates of the magnitude of morbidity and mortality related to the health-related event under surveillance, including the identification of factors associated with the event?
detect trends that signal changes in the occurrence of disease, injury, or adverse or protective exposure, including detection of epidemics (or outbreaks)?
permit assessment of the effect of prevention and control programs?
lead to improved clinical, behavioral, social, policy, or environmental practices? or
stimulate research intended to lead to prevention or control?

amount and type of data necessary to establish that the health-related event has occurred (i.e., the case definition has been met);
amount and type of other data on cases (e.g., demographic, behavioral, and exposure information for the health-related event);
number of organizations involved in receiving case reports;
level of integration with other systems;
method of collecting the data, including number and types of reporting sources, and time spent on collecting data;
amount of follow-up that is necessary to update data on the case;
method of managing the data, including time spent on transferring, entering, editing, storing, and backing up data;
methods for analyzing and disseminating the data, including time spent on preparing the data for dissemination;
staff training requirements; and
time spent on maintaining the system.

special or follow-up laboratory tests to confirm the case;
investigation of the case, including telephone contact or a home visit by public health personnel to collect detailed information;
multiple levels of reporting (e.g., with the National Notifiable Diseases Surveillance System, case reports might start with the health-care provider who makes the diagnosis and pass through county and state health departments before going to CDC [29]); and
integration of related systems whereby special training is required to collect and/or interpret data.

subject or agency participation rate (if it is high, how quickly it was achieved);
interview completion rates and question refusal rates (if the system involves interviews);
completeness of report forms;
physician, laboratory, or hospital/facility reporting rate; and
timeliness of data reporting.

the public health importance of the health-related event;
acknowledgment by the system of the person's contribution;
dissemination of aggregate data back to reporting sources and interested parties;
responsiveness of the system to suggestions or comments;
burden on time relative to available time;
ease and cost of data reporting;
federal and state statutory assurance of privacy and confidentiality;
the ability of the system to protect privacy and confidentiality;
federal and state statute requirements for data collection and case reporting; and
participation from the community in which the system operates.

certain diseases or other health-related events are occurring in the population under surveillance;
cases of certain health-related events are under medical care, receive laboratory testing, or are otherwise coming to the attention of institutions subject to reporting requirements;
the health-related events will be diagnosed/identified, reflecting the skill of health-care providers and the sensitivity of screening and diagnostic tests (i.e., the case definition); and
the case will be reported to the system.

the number of persons who have telephones, who are at home when the call is placed, and who agree to participate;
the ability of persons to understand the questions and correctly identify their status; and
the willingness of respondents to report their status.

conducting active surveillance (i.e., contacting all providers and institutions responsible for reporting cases);
using external standards (or other surveillance indicators) to monitor the quality of case reporting;
identifying imported cases;
tracking the number of cases of suspected disease that are reported, investigated, and ruled out as cases;
monitoring the diagnostic effort (e.g., tracking submission of laboratory requests for diagnostic testing); and
monitoring the circulation of the agent (e.g., virus or bacterium) that causes the disease.

characteristics of the population, including, age, socioeconomic status, access to health care, and geographic location (60);
clinical course of the disease or other health-related event (e.g., latency period, mode of transmission, and outcome [e.g., death, hospitalization, or disability]);
prevailing medical practices (e.g., sites performing diagnostic tests and physicianreferral patterns) (33,61); and
multiple sources of data (e.g., mortality rates for comparison with incidence data and laboratory reports for comparison with physician reports).

the number of unscheduled outages and down times for the system's computer;
the costs involved with any repair of the system's computer, including parts, service, and amount of time required for the repair;
the percentage of time the system is operating fully;
the desired and actual amount of time required for the system to collect or receive data;
the desired and actual amount of time required for the system to manage the data, including transfer, entry, editing, storage, and back-up of data; and
the desired and actual amount of time required for the system to release data.

SUMMARY

The guidelines in this report address evaluations of public health surveillance systems. However, these guidelines could also be applied to several systems, including health information systems used for public health action, surveillance systems that are pilot tested, and information systems at individual hospitals or health-care centers. Additional information can also be useful for planning, establishing, as well as efficiently and effectively monitoring a public health surveillance system (6--7). To promote the best use of public health resources, all public health surveillance systems should be evaluated periodically. No perfect system exists; however, and tradeoffs must always be made. Each system is unique and must balance benefit versus personnel, resources, and cost allocated to each of its components if the system is to achieve its intended purpose and objectives. The appropriate evaluation of public health surveillance systems becomes paramount as these systems adapt to revised case definitions, new health-related events, new information technology (including standards for data collection and sharing), current requirements for protecting patient privacy, data confidentiality, and system security. The goal of this report has been to make the evaluation process inclusive, explicit, and objective. Yet, this report has presented guidelines --- not absolutes --- for the evaluation of public health surveillance systems. Progress in surveillance theory, technology, and practice continues to occur, and guidelines for evaluating a surveillance system will necessarily evolve.

References

CDC. Guidelines for evaluating surveillance systems. MMWR 1988;37(No. S-5).
Health Information and Surveillance System Board. Integration Project: National Electronic Disease Surveillance System. Available at . Accessed May 7, 2001.
Department of Health and Human Services. Administrative simplification. Available at . Accessed May 7, 2001.
CDC. Framework for program evaluation in public health. MMWR 1999;48(RR-11).
Thacker SB. Historical development. In: Teutsch SM, Churchill RE, eds. Principles and practice of public health surveillance, 2nd ed. New York, NY: Oxford University Press, 2000.
Buehler JW. Surveillance. In: Rothman KJ, Greenland S. Modern epidemiology, 2nd ed. Philadelphia, PA: Lippencott-Raven, 1998.
Teutsch SM, Thacker SB. Planning a public health surveillance system. Epidemiological Bulletin: Pan American Health Organization 1995;16:1--6.
Thacker SB, Stroup DF. Future directions for comprehensive public health surveillance and health information systems in the United States. Am J Epidemiol 1994;140:383--97.
Yasnoff WA, O'Carroll PW, Koo D, Linkins RW, Kilbourne EM. Public health informatics: improving and transforming public health in the information age. J Public Health Management Practice 2000;6:63--71.
CDC. An ounce of prevention: what are the returns? 2nd ed. Atlanta, GA: Department of Health and Human Services, CDC, 1999.
CDC. Impact of vaccines universally recommended for children---United States, 1990--1998. MMWR 1999;48:243--8.
Hinman AR, Koplan JP. Pertussis and pertussis vaccine: reanalysis of benefits, risks, and costs. JAMA 1984;251:3109--13.
Institute of Medicine, Committee on Summary Measures of Population Health. In: Field MJ, Gold MR, eds. Summarizing population health: directions for the development and application of population metrics. Washington, DC: National Academy Press, 1998. Available at
Dean AG, West DJ, Weir WM. Measuring loss of life, health, and income due to disease and injury. Public Health Rep 1982;97:38--47.
Vilnius D, Dandoy S. A priority rating system for public health programs. Public Health Rep 1990;105:463--70.
CDC. Case definitions for infectious conditions under public health surveillance. MMWR 1997;46(RR-10).
Council of State and Territorial Epidemiologists. Position statements. Available at . Accessed May 7, 2001.
Morris G, Snider D, Katz M. Integrating public health information and surveillance systems. J Public Health Management Practice 1996;2:24--7.
Meriwether RA. Blueprint for a National Public Health Surveillance System for the 21st century. J Public Health Management Practice 1996;2(4):16--23.
Zachman JA. A framework for information systems architecture. IBM Systems J 1987;26(3).
Sowa JF, Zachman JA. Extending and formalizing the framework for information systems architecture. IBM Systems J 1992;31(3).
Health Information and Surveillance System Board. Public Health Conceptual Data Model (PHCDM). Available at . Accessed May 7, 2001.
Koo D, Parrish RG II. The changing health-care information infrastructure in the United States: opportunities for a new approach to public health surveillance. In: Teutsch SM, Churchill RE, eds. Principles and practice of public health surveillance, 2nd ed. New York, NY: Oxford University Press, 2000.
Data Interchange Standards Association. X12 Standards, release 4020. Alexandria, VA: Accredited Standards Committee X12, 1998. Available at . Accessed May 7, 2001.
Health Level Seven. Available at . Accessed May 7, 2001.
Health Care Financing Administration. The Health Insurance Portability and Accountability Act of 1996 (HIPAA). Available at . Accessed May 7, 2001.
Humphreys BL, Lindberg DAB, Schoolman HM, Barnett GO. The unified medical language system: an informatics research collaboration. JAMIA 1998;5:1--11.
College of American Pathologists. Systematized nomenclature of human and veterinary medicine (SNOMED � ), version 3.5. Northfield, IL: College of American Pathologists. Available at . Accessed May 7, 2001.
Koo D, Wetterhall SF. History and current status of the National Notifiable Diseases Surveillance System. J Public Health Management Practice 1996;2:4--10.
Council of State and Territorial Epidemiologists. Data release guidelines of the Council of State and Territorial Epidemiologists for the National Public Health Surveillance System. Atlanta, GA: Council of State and Territorial Epidemiologists, June 1996. Available at . Accessed February 2001.
Privacy Law Advisory Committee, Model State Public Health Privacy Project. Model state public health privacy act. Washington, DC: Georgetown University Law Center, 1999. Available at . Accessed May 7, 2001.
Federal Committee on Statistical Methodology, Subcommittee on Disclosure Limitation Methodology. Statistical Policy Working Paper 22: report on statistical disclosure limitation methodology. Washington, DC: Office of Management and Budget, May 1994 (PB94-165305). Available at . Accessed May 7, 2001.
Vogt RL, LaRue D, Klaucke DN, Jillson DA. Comparison of an active and passive surveillance system of primary care providers for hepatitis, measles, rubella, and salmonellosis in Vermont. Am J Public Health 1983;73:795--7.
Hinds MW, Skaggs JW, Bergeisen GH. Benefitcost analysis of active surveillance of primary care physicians for hepatitis A. Am J Public Health 1985;75:176--7.
Morris S, Gray A, Noone A, Wiseman M, Jathanna S. The costs and effectiveness of surveillance of communicable disease: a case study of HIV and AIDS in England and Wales. J Public Health Med 1996;18:415--22.
Haddix AC, Teutsch SM, Shaffer PA, Du�et, DO, eds. Prevention effectiveness: a guide to decision analysis and economic evaluation. New York, NY: Oxford University Press, 1996.
Department of Health and Human Services, Health Resources and Services Administration. Title V (Title V IS) information system web site. Accessed at . Accessed May 7, 2001.
US Department of Health and Human Services. Healthy people 2010 (conference ed, 2 vols). Washington, DC: US Department of Health and Human Services, 2000.
Behavioral Risk Factor Surveillance System. Available at . Accessed May 7, 2001.
Klevens RM, Fleming PL, Neal JJ, Mode of Transmission Validation Study Group. Is there really a heterosexual AIDS epidemic in the United States? Findings from a multisite validation study, 1992-1995. Am J Epidemiol 1999;149:75--84.
Fox J, Stahlsmith L, Remington P, Tymus T, Hargarten S. The Wisconsin firearm-related injury surveillance system. Am J Prev Med 1998;15:101--8.
Phillips-Howard PA, Mitchell J, Bradley DJ. Validation of malaria surveillance case reports: implications for studies of malaria risk. J Epidemiol Community Health 1990;44:155--61.
Weinstein MC, Fineberg HV. Clinical decision analysis. Philadelphia, PA: W.B. Saunders, 1980:84--94.
CDC. Manual for the surveillance of vaccine-preventable diseases. Atlanta, GA: Department of Health and Human Services, CDC, September 1999.
Bender JB, Hedberg CW, Besser JM, MacDonald KL, Osterholm MT. Surveillance for Escherichia coli 0157:H7 infections in Minnesota by molecular subtyping. N Engl J Med 1997;337:388--94.
Chandra Sekar C, Deming WE. On a method of estimating birth and death rates and the extent of registration. J Am Stat Assoc 1949;44:101--15.
Emori TG, Edwards JR, Culver DH, et al. Accuracy of reporting nosocomial infections in intensive-care--unit patients to the National Nosocomial Infections Surveillance System: a pilot study. Infect Control Hosp Epidemiol 1998;19:308--16.
Johnson RL, Gabella BA, Gerhart KA, McCray J, Menconi JC, Whiteneck GG. Evaluating sources of traumatic spinal cord injury surveillance data in Colorado. Am J Epidemiol 1997;146:266--72.
Watkins ML, Edmonds L, McClearn A, Mullins L, Mulinare J, Khoury M. The surveillance of birth defects: the usefulness of the revised US standard birth certificate. Am J Public Health 1996;86:731--4.
Payne SMC, Seage GR III, Oddleifson S, et al. Using administratively collected hospital discharge data for AIDS surveillance. Ann Epidemiol 1995;5:337--46.
Van Tuinen M, Crosby A. Missouri firearm-related injury surveillance system. Am J Prev Med 1998;15:67--74.
Hook EB, Regal RR. The value of capture-recapture methods even for apparent exhaustive surveys. Am J Epidemiol 1992;135:1060--7.
Gazarian M, Williams K, Elliott E, et al. Evaluation of a national surveillance unit. Arch Dis Child 1999;80:21--7.
Singh J, Foster SO. Sensitivity of poliomyelitis surveillance in India. Indian J Pediatr 1998;35:311--5.
Last JM, Abramson JH, Friedman GD, Porta M, Spasoff RA, Thuriaux M, eds. A dictionary of epidemiology, 3rd ed. New York, NY: Oxford University Press, 1995:173--4.
German RR. Sensitivity and predictive value positive measurements for public health surveillance systems. Epidemiology 2000;11:720--7.
Hedegaard H, Wake M, Hoffman R. Firearm-related injury surveillance in Colorado. Am J Prev Med 1998;15(3S):38--45.
M�h�nen M, Salomaa V, Brommels M, et al. The validity of hospital discharge register data on coronary heart disease in Finland. Eur J Epidemiol 1997;13:403--15.
LeMier M, Cummings P, Keck D, Stehr-Green J, Ikeda R, Saltzman L. Washington state gunshot-wound surveillance system. Am J Prev Med 1998;15(3S):92--100.
Kimball AM, Thacker SB, Levy ME. Shigella surveillance in a large metropolitan area: assessment of a passive reporting system. Am J Public Health 1980;70:164--6.
Thacker SB, Redmond S, Rothenberg RB, Spitz SB, Choi K, White MC. A controlled trial of disease surveillance strategies. Am J Prev Med 1986;2:345--50.
McClure RJ, Burnside J. The Australian Capital Territory Injury Surveillance and Prevention Project. Acad Emerg Med 1995;2:529--34.
US Bureau of the Census. Federal-State Cooperative Program for Population Estimates: Operational Guidelines. Washington, DC: US Bureau of the Census, July 1992. Available at . Accessed February 2001.
Hahn RA, Stroup DF. Race and ethnicity in public health surveillance: criteria for the scientific use of social categories. Public Health Rep 1994;109:7--15.
CDC. New population standard for age-adjusting death rates. MMWR 1999;48:126--7.
Alter MJ, Mares A, Hadler SC, Maynard JE. The effect of underreporting on the apparent incidence and epidemiology of acute viral hepatitis. Am J Epidemiol 1987;125:133--9.
Romaguera RA, German RR, Klaucke DN. Evaluating public health surveillance. In: Teutsch SM, Churchill RE, eds. Principles and practice of public health surveillance, 2nd ed. New York, NY: Oxford University Press, 2000.
Rosenberg ML. Shigella surveillance in the United States, 1975. J Infect Dis 1977;136:458--60.
CDC. Preventing lead poisoning in young children: a statement by the Centers for Disease Control---October 1991. Atlanta, GA: Department of Health and Human Services, Public Health Service, CDC, 1991.
Maiese DR. Data challenges and successes with healthy people. Hyattsville, MD: Department of Health and Human Services, CDC, National Center for Health Statistics. Healthy People 2000 Statistics and Surveillance 1998. (no. 9).
Effler P, Ching-Lee M, Bogard A, Ieong M-C, Nekomoto T, Jernigan D. Statewide system of electronic notifiable disease reporting from clinical laboratories: comparing automated reporting with conventional methods. JAMA 1999;282:1845--50.
Yokoe DS, Subramanyan GS, Nardell E, Sharnprapai S, McCray E, Platt R. Supplementing tuberculosis surveillance with automated data from health maintenance organizations. Emerg Infect Dis 1999:5:779--87.
Johnson AM Jr, Malek M. Survey of software tools for evaluating reliability, availability, and serviceability. Association for Computing Machinery Surveys 1988;20(4).
Snider DE, Stroup DF. Ethical issues. In: Teutsch SM, Churchill RE, eds. Principles and practice of public health surveillance, 2nd ed. New York, NY: Oxford University Press, 2000.
Singleton JA, Lloyd JC, Mootrey GT, Salive ME, Chen RT, VAERS Working Group. An overview of the vaccine adverse event reporting system (VAERS) as a surveillance system. Vaccine 1999;17:2908--17.

Disclaimer All MMWR HTML versions of articles are electronic conversions from ASCII text into HTML. This conversion may have resulted in character translation or format errors in the HTML version. Users should not rely on this HTML document, but are referred to the electronic PDF version and/or the original MMWR paper copy for the official text, figures, and tables. An original paper copy of this issue can be obtained from the Superintendent of Documents, U.S. Government Printing Office (GPO), Washington, DC 20402-9371; telephone: (202) 512-1800. Contact GPO for current prices.

**Questions or messages regarding errors in formatting should be addressed to mmwrq@cdc.gov.

Morbidity and Mortality Weekly Report
Centers for Disease Control and Prevention
1600 Clifton Rd, MailStop E-90, Atlanta, GA 30333, U.S.A

This page last reviewed 7/19/2001