Panel providers are asked for the "active panel size" of their panel using 2 different formats. First, providers are asked for their panel size according to their internal definition of an active panelist. If they have no internal definition, MRA Review Program requests that they provide a panel size and definition that includes those database members who are regularly invited to complete surveys. It is this internal definition that is thought to be most meaningful to describe actual practices, and it is used to calculate panel attrition and tenure.

Secondly, providers are asked to estimate their panel size according to the ISO active panel definition: A panel member that has participated in at least one survey, or updated his/her profile data, or registered to join panel, within last 12 months. This provides a standardized number to use when comparing the relative sizes of one panel to another.

Why is this important?
The active panel size feeds into other critical panel metrics. Additionally, it describes the basic capacity of the panel to meet the needs of a research project.

Average amount of time active panelists have been members of panel, in months. This is reported in mean, median, and mode.

Why is this important?
The average length of panel membership may reflect the panel provider's relationship with their panel, the experience (and potential maturation) of panelists, and also serve as an indicator of their philosophy toward panel maintenance.

Information describing policy of category exclusion. Category exclusion refers to limitations placed on panelists participating in surveys of a similar nature within a given period of time. Not all panel providers maintain a policy of category exclusion.

Why is this important?
Exposure to information may or may not alter a panelist's opinions. Category exclusion seeks to limit panelist's exposure to similar subjects in order to help ensure their opinions accurately represent the opinions of others, who are not exposed to such information.

Description of metrics offered by panel provider that indicate level of non-response (or participation). Panel provider also notes whether information is "standard" or "upon request."

Why is this important?
Details of the amount of non-response may be critical to understanding threats of non-response error as well as other performance details of research project.

Panels lose members over time, voluntarily and involuntarily. This field describes the panel provider's policy on whether they remove members and what criteria trigger the removal of panelists.

Why is this important?
Panel providers each have unique policies on how to maintain high quality panels. Attrition protocols help to describe how panelists are maintained as well as whether certain behaviors and/or qualities are actively managed. A comparison of attrition policies should help to shed light on the overall standards each panel maintains in terms of the performance of their member panelists. It is also helpful in providing context for the attrition rate statistic.

Panel attrition is defined and calculated as the percentage of panelists removed from active panelist status over the period of one year. Panel providers calculate this statistic based upon the previous year's figures. Thus, the attrition statistic reflects panel attrition for the previous year

Panel Attrition Formula
PCurrent= # of active panelists as of January 1st of current year.
Attprv= # of active panelists becoming inactive, voluntarily attrited, or involuntarily attrited throughout previous year.
Forumla: Attrition = (Attprv)/ ((PCurrent) + (Attprv))

Why is this important?
Attrition may be a key indicator of panel management. Panels may attrite panel members deliberately, or may lose them when panelists decide to become inactive or resign from membership. It is important to view the panel attrition statistic alongside each panel's policies for attriting members and other important panel characteristics.

For example, a high attrition rate due to respondent dissatisfaction may be an indication of poor panel management. However, panels that maintain strict standards of respondent performance may tend to attrite more panel members from their database.

A description of what the panel provider believes makes their panel unique. Here the panel provider describes the unique strengths and advantages that their panel offers to the MRA Review Program website audience.

Why is this important?
The MRA Review Program reporting system is based on factors that experts throughout the survey research profession have identified as being important to data quality. However, each panel provider may have unique solutions and approaches which are not commonly discussed among survey research professionals. For this reason, it's important to weigh the standardized information provided by MRA Review Program reports along with those features identified by panel providers in the "Panel Differentiation" section.

A description of what the panel provider believes makes their panel unique. Here the panel provider describes the unique strengths and advantages that their panel offers to the MRA Review Program website audience.

Why is this important?
The MRA Review Program reporting system is based on factors that experts throughout the survey research profession have identified as being important to data quality. However, each panel provider may have unique solutions and approaches which are not commonly discussed among survey research professionals. For this reason, it's important to weigh the standardized information provided by MRA Review Program reports along with those features identified by panel providers in the "Panel Differentiation" section.

Panels will naturally turnover members from their base, and will need to replenish their membership over time. This field provides information about the panel provider's policy on panel refreshment.

Why is this important?
The frequency and regularity of panel refreshment may or may not affect reliability (sometimes called "repeatability") and validity. Like most data quality issues in online panel research, the effect of panel refreshment policies on data quality is in need of basic research. However, theory suggests that refreshment policies may play a role in the quality of research findings.

For example, a sudden influx of new panelists may change the composition of the panel so that the membership is abruptly composed of a higher proportion of panelists with little-to-no survey experience. This may, or may not, affect survey results.

Tracking studies may be particularly sensitive to such changes, where survey estimates may fluctuate due to the panelists experience in survey research, as opposed to a general change in opinion or behavior. It is important to consider the policy a panel sustains in order to maintain a balance of new and/or seasoned panelists.

A description of the make-up of the panel. Also included is a description of the attributes the panel is able to screen for as well as a general description of the sources from which panelists are recruited.

Why is this important?
This report item may be used to better understand whether not the panel fits with the type of respondents you need to interview. The "panel type" field helps you to determine which panels are able to provide the types of respondents you need to complete your research project.

Some panels are used explicitly for research; others may be used for research and other purposes. This field indicates the extent of how the database of panel members is utilized.

Why is this important?
Online access panels are developed with a focus on managing a relationship with their members. The ways in which members are utilized may not only affect their relationship with the panel provider, but will also expose them to a variety of experiences that may alter their attributes and/or personal qualities.

Although most panels will be used only for research purposes, some panels may also utilize panelists for other activities. It is important to consider whether additional uses may affect panelists responses in research.

Description of protocols panel provider uses to ensure legitimacy of panelists. This includes verifying information about panelists within their database.

Why is this important?
Verification processes are thought to be important safeguards against fraudulent panelists. Fraudulent panelists may seek and join panels in order to reap incentive rewards, or for other reasons, and may falsely represent their identities or other personal characteristics, qualities, or attributes. The validation protocols listed in the MRA Review Program system help to paint a picture of the provider's overall plan to ensure occurrences of fraudulent (or otherwise undesirable) respondents are minimized.

Description of the maximum number of times panelists are permitted to participate in research in a given period of time, as well as the average number of times panelists have participated in a survey (who have completed at least one project within the last 30 days.) Not all panel providers may maintain participation limits.

Why is this important?
Conventional wisdom posits that the ability to frequently participate in research potentially increases occurrences of disingenuous, satisificing, and other undesirable behaviors or data quality results. This is because frequent participation may yield potential for high monetary rewards and encourage respondents to participate who are either overly motivated by incentives or who are significantly unique in other important ways.

Many organizations hold limitations on the number of times a panel member may participate in a survey per a give time period. They may also forgo this rule under specific circumstances. It is important to understand the provider's policy toward frequency of panelist participation.

Information describing pre-testing policies. Notations are provided for whether or not pre-testing is available as a standard feature, or upon request. Pre-testing categories include, but are not limited to:

• Content Quality assurance review process of questionnaire/research design
• Programming Quality assurance review process of programming functions
• Grammar & Spelling Quality assurance review process of grammar and spelling
• Timing Quality assurance review process of time to completion
• Design & Formatting&Quality assurance review process of design and formatting

Why is this important?
Pre-testing can serve as a powerful quality assurance in the survey process. However, it is important to consider the different processes that are included in the vendor's pre-testing offerings.

Information describing organizational privacy policy and data safeguards. A description of privacy policy, policies regarding interviewing minors, and opt-in policies are included., Notation is also provided to indicate:

• Whether panel provider's privacy policy accessible from corporate website
• Whether privacy policy is linked to from every survey instrument.
• Whether all employees that are involved in data handling/collection are trained in privacy policy.
• Whether a designated person within organization serves as internal point of contact for data privacy issues.
• Whether a designated person within organization serves as external point of contact for data privacy issues.
• Whether organization offers an automated opt-out in all emails to respondents.
• Whether organization maintains an internal do-not-contact list.

Why is this important?
A panel provider's privacy policy is important in establishing the nature of a panelist's participation as well as in describing the legal considerations that provider has in place.

A description of information collected in the panelist's profile. Also indicated is the availability of this information for different purposes (i.e. collected in database, available for screening, available as variables in dataset)

Why is this important?
Information attached to panelists may be vital to the design of a research project, for screening purposes, reporting purposes, and even for other uses, such as an analysis of non-response error.

Description of protocols panel provider uses to ensure legitimacy of panelists participating in survey project. This includes verifying information about panelists who are participating in research projects, and may include their own panelists as well as others who are directed from supplemental sources to meet the needs of the project. The validation protocols listed in the MRA Review Program system help to paint a picture of the provider's overall plan to ensure occurrences of fraudulent (or otherwise undesirable) respondents are minimized.

Why is this important?
Verification processes are thought to be important safeguards against fraudulent panelists. Fraudulent panelists may seek and join panels in order to reap incentive rewards, or for other reasons, and may falsely represent their identities or other personal characteristics, qualities, or attributes.

Indication of whether examples of recruiting materials are available upon request.

Why is this important?
The initial communication/message that a potential panel member recruit receives may provide their motivation for joining. This may be important when considering your project needs and how respondent motivations link with what you are researching.

For example, recruitment materials that strongly emphasize monetary incentives as a benefit of membership may or may not attract panelists primarily motivated to participate in research in order to reap monetary rewards and who may also be more prone to satisficing behaviors. This could be problematic depending upon how the panel provider manages their panelists, prevents fraudulent members, investigates satisficers, etc.

Likewise, panelists motivated by other pieces of information told to them through the recruiting process may be inclined to participate for reasons linked with that information. If the panel advertises specific survey content for example, a promotional advertisement for taking surveys about cars, they may yield an unusually large number of car enthusiasts in their database. This may or may not affect the results of your research project, depending on whether respondent motivations for joining link with the concepts/attributes that you are interested in studying.

Considering the types of panelists and their motivations for joining a panel may help reduce sources of error in your project.

Description of sampling protocols and degree of flexibility in sampling process.

Why is this important?
Selection, quota, and weighing protocols must be designed and implemented appropriately in order to ensure that the data most accurately reflects the target population of interest. This field describes standard protocols and indicates whether the panel purchaser has flexibility in controlling the sampling plan and adjustments.

Description of protocols panel provider uses to ensure that panelists/respondents are participating in an engaged manner, using an appropriate level of cognition to answer survey questions.

Why is this important?
A fundamental source of error and threat to data quality can ensue when respondents do not participate in an engaged manner. Sometimes respondents may not give an appropriate level of effort while reflecting on survey questions, other times, respondents may avoid considering survey questions altogether- simply choosing survey response options at will.

Description of information provided at the close of the research project. This includes data that relates to the performance of the project, but does not include measures of non-response. Panel provider also notes whether information is "standard" or "upon request."

Why is this important?
Details of the performance of a research project are critical to understanding the data garnered from the project.

Description of survey invitation protocols and degree of flexibility in adapting survey invitation

Why is this important?
Information given to respondents during invitation may affect their responses to survey questions as well as influence their motivations for participation. Consideration should be placed on how mandatory elements may interact with research design.

Panels may use respondents provided from external panels or sources. In this field, the panel provider indicates whether or not they ever supplement their samples with third-party supplements, and if so, 1) whether advanced notification is given to panel purchases in such cases and 2) whether the panel purchaser is informed of the names of the organizations/sources supplementing the research project.

Why is this important?
The panel reports published in this system offer a comprehensive look at important factors in panel management and processes. However, if third-party respondents are being used, it is important to consider the quality of the organizations that are supplying them. Notably, there is an additional consideration of how panel addresses the potential threat of duplicate panelists from the third-party.

The "third-party supplements" field informs MRA Review Program users of the transparency they should expect regarding their vendor's use of external panel suppliers.