Now, let's get started!
Certified reviewers who meet the necessary security requirements will receive a link
to access the contract SRG.
Alternatively, if you have been assigned a review task, you can click on the link in
the automatically generated task assignment email you received.
If you are logging in to the SRG for the first time, follow the instructions you received
by email to get started.
Then, enter your username and password to log in.
Once you log in to the contract SRG, you will be taken to the My Tasks page.
This page lists all reviews currently assigned to you along with the relevant protocol, review
status, and due date.
To begin a review, click on the Review button next to the study on the My Tasks page.
You can access the public version of the SRG on the WWC website.
If you are starting a new review, select "Start new review."
You will need to enter the study citation, and select the intervention being studied,
the version of the WWC design standards you are using for the review, the review protocol,
and the version of the review protocol you are using.
Once you have completed those fields, select Save.
When you select Save, the public SRG will generate a PIN for this review and prompt
you to write down this PIN or receive it via email.
You will need this PIN to access the review in the future.
If you are continuing an existing review, select Enter PIN on the public SRG landing
page and enter the automatically generated PIN to access the review.
Now we'll go through each stage of the review process.
The first SRG stage is the screening stage.
During this stage, you will answer a series of questions to determine if a study is eligible
for WWC review under the relevant review protocol.
At the top of the screening page, you will see a brief description of the intervention
the study evaluated.
Following the intervention name and description, there are 11 Yes or No screening questions.
Starting with Question 1, click Yes if the study qualifies as a study under the WWC definition.
Click Yes again for Question 2 if the study's primary analysis examines the effect of an
intervention.
Continue answering each of the screening questions by selecting Yes or No.
You will notice that some questions have italicized text beneath them with additional eligibility
information.
This text comes from the relevant review protocol.
You can use this text as guidance while you are answering the questions.
For example, Question 6 specifies that, according to the Beginning Reading review protocol,
the eligible age or grade range is kindergarten through 3rd grade or ages 5 through 8, if
the grade is not specified.
Based on the answers you provide, the SRG will display whether the study is eligible
for review.
Before we continue to the next review stage, let's discuss how to navigate through the
SRG and how to document any additional information, concerns, or questions you may have about
the study besides the information the SRG specifically requests.
At the bottom of each page, you will see two options: "Save" and "Save and Continue."
The Save option simply saves the information you have entered but keeps you on the same
screen.
The Save and Continue option saves the information you entered and moves you to the next step
in the review process.
In this case, it moved us to the Tags stage.
You can navigate to previous pages by either selecting the Previous button or clicking
on the stage at the top of the screen.
In the bottom, right-hand corner of most pages, you will see a Notes button.
This is where you can document any concerns or questions you have about the study that
are not documented elsewhere in the SRG, and you can note any assumptions you made during
your review.
For example, you can include details about the study design, especially any that affect
the review.
Also, describe any information not reported in the study that is needed to complete the
review.
For contract reviews, the WWC will request this information from authors before completing
the review.
There is a separate notes field available on the Measures page that allows you to document
notes specific to an outcome measure.
As a reminder, it is a best practice—and a requirement for contract reviewers—to
use complete sentences in the notes and include page references where needed.
After you enter your note, click Save.
Then you can select Hide Notes to hide the notes screen.
You will be able to open and update the study notes from other pages during the review process.
Once you click Save and Continue, the SRG will move to the Tag stage.
During this stage, you will select the WWC topics that are relevant to the study you
are reviewing.
As a reminder, this stage is only available in the contract SRG version.
In our case, we will select "Literacy" and "Kindergarten to 12th Grade" because
we are reviewing a study that evaluates a reading intervention in elementary schools.
Information entered during this stage in the contract version will determine how the study
is categorized for searching and filtering on the WWC website's Find What Works page.
After you select the Tags, click Save and Continue.
The Measures stage is where you will enter the information about each eligible outcome
measure that determines whether a finding meets WWC design standards and, ultimately,
whether the entire study meets WWC design standards.
In the SRG, an outcome measure is defined by the instrument or method used to assess
the outcome, and also by the timing of follow-up and the sample evaluated.
For example, a study may assess graduation rates as the only outcome measure, but you
may need to enter multiple measures into the SRG if graduation rates are measured for the
full sample and for subsamples of students, or if they are measured at both four-year
and five-year follow-ups.
You should enter each combination of a sample and follow-up time point into the SRG as a
separate measure.
You only need to report information for eligible outcome measures.
Eligibility is determined by the review protocol, which describes the outcome domains that are
eligible for review under that protocol.
To add an outcome measure, select "Add new outcome."
Use this page to enter information about each measure used in the study.
First, select a measure.
To do this, search for the measure by the measure name.
For example, we will enter "Alphabetics" as though we were searching for the measure
"Sample Alphabetics Assessment."
If you don't find a measure, try alternate spellings or key words to ensure that the
measure does not already exist.
"Sample Alphabetics Assessment" is one of the search results that comes up.
We'll check that the measure is in an eligible domain for our review protocol and then we'll
select that measure.
Sometimes the measure will be new to the WWC database and you will need to create a new
measure.
You will still start by searching for the measure.
This time, let's search for Oklahoma's standardized reading test by searching for
"Oklahoma."
Since no results appear, the SRG offers the option to add it as a new measure.
To do this, select "Add a new measure" from the search page.
On the Add Measure page, select the appropriate domain for the measure.
Enter the measure name.
Double-check that the measure name is spelled correctly.
Note that when you click on a field, the SRG will provide additional guidance on how to
input information, such as "Provide the full name of measure or test" for the measure
name field.
Add a brief description of the measure, again checking your spelling and grammar.
Now you will need to answer a series of questions to determine if the measure meets the WWC
validity and reliability requirements for review discussed in the Outcome Measures module,
which is Module 5.
Here and elsewhere in the SRG, you can click on the question marks beside a question for
more information on how to answer that question.
If you do not have information on test-retest reliability, internal consistency, or inter-rater
reliability for a standardized test, you can leave those spaces blank.
When you are done entering information about a measure, select Save.
Now you will answer additional questions to determine if the measure meets review requirements
and begin to review the study against WWC group design standards.
The first two questions ask whether the measure is overaligned or if there is a data collection
confounding factor.
Refer to the Outcomes Measures module, which is Module 5, for additional information on
answering these questions.
The third question asks if the analysis controlled for any endogenous covariates.
This potential issue with the analytic approach is discussed in Chapter II, Section C of the
WWC Standards Handbook.
Next, enter the follow-up period for the measure.
This is the time elapsed between the end of the intervention and the assessment of the
measure.
For our example, we'll say that the reading achievement assessment was administered to
students six months after the intervention ended.
If the follow-up was conducted immediately after the intervention ended, or you do not
know how much time elapsed, enter zero and select "Days."
The next set of questions ask about the study design.
First, select whether the study is a randomized controlled trial (that is, an RCT) or a quasi-experimental
design study (a QED).
If you select randomized controlled trial, you will be asked whether the randomized controlled
trial was compromised.
For our example, we'll say that this study is a randomized controlled trial that was
not compromised.
The next question will ask whether the study includes any confounding factors for this
measure.
You can refer to the Group Designs module, which is Module 1, for more information on
determining the study design and on compromised randomized controlled trials, and the Confounding
Factors module, which is Module 4.
Finally, select an appropriate description of the comparison group.
If the intervention of interest was compared against another intervention, you can select
Intervention and then select the specific intervention.
Other options include Business as Usual, None, and Unknown.
In the next section, you will describe the study sample.
As we mentioned earlier, the sample is part of how the SRG defines a measure.
For example, you would enter the reading assessment outcome for the study's full sample as a
separate measure from the reading assessment outcome for a subsample of grade 1 students.
If you select Yes for the question "Do these data represent the full sample?," the Sample
Description box will automatically populate with "Full sample."
If you select No, you will see several common subsample options.
For example, you might select Gender and then Male to indicate that this is a subsample
of boys from the full study sample.
The subsample information you select will automatically display in the Sample Description
box.
Let's say that your subsample doesn't fit into any of these categories.
In that case, enter a description of the sample in the Sample Description box.
For our example, we will indicate that this finding is for the full sample.
Next, you will enter information about the study's sample sizes.
First, answer Yes or No to whether the measure is based on a cluster-level assignment study
design.
If you select Yes, the SRG will ask different questions about missing data, the sample sizes
for the clusters and individuals, and how the analysis was conducted and outcomes were
reported.
For more information, refer to the Cluster-Level Assignment module which is Module 8.
We will select No for our example, indicating the measure is based on an individual-level
assignment study design.
Because this is not a cluster-level assignment study, the next two questions ask about missing
data.
First, the SRG will ask whether outcome data were imputed for this measure.
If you select Yes, the SRG will ask for some additional sample sizes.
Second, the SRG will ask whether the study used an acceptable approach for addressing
missing data.
For more information on imputation and acceptable approaches for addressing missing data, refer
to Module 2 on Attrition, Module 3 on Baseline Equivalence, and the WWC Standards Handbook.
Finally, you will enter the sample sizes for the study.
This is the information the SRG will use to calculate study attrition for a randomized
controlled trial.
First, select the unit that describes the individuals in the study.
Next, enter the number of units randomly assigned to the intervention and comparison conditions.
Then, enter the analytic sample size for each condition.
If you do not know the sample sizes, you can also select Not Reported under each box, and
include a note that the WWC may need to request this information from the author.
Now, you'll enter information needed to determine whether the study satisfies the
baseline equivalence requirement.
For information about this requirement, refer to the Baseline Equivalence module, which
is Module 3.
Although this study is a low-attrition RCT that does not need to satisfy the baseline
equivalence requirement, reviewers should always enter data on the baseline measures
in the study.
To start, select "Add a baseline measure."
Choose "Select measure" to choose the baseline equivalence measure.
The contract SRG will again provide a list of measures, or the reviewer can enter a new
measure.
Imagine that the study reports a measure of family income at baseline.
First, we'll type "income" to see if a measure of family income already exists.
It does, so we'll select that measure from the list.
Next, we'll answer the question on whether this baseline measure is required for establishing
baseline equivalence.
You can determine this by referring to the review protocol, which lists measures required
for baseline equivalence.
In our example, we will say that this is not required for baseline equivalence because
socioeconomic status is not required under the Beginning Reading review protocol.
Next, we will answer three questions about the baseline data, referring to Modules 3
and 8 or the WWC Standards Handbook, as needed.
Then, we will enter the baseline sample sizes and baseline mean scores for the intervention
and comparison groups.
If this information is not available, select "Not Reported."
Note that if we were entering a dichotomous variable, such as the percentage of male students,
we would enter the means as percentages without the percent symbol and without converting
it into a proportion.
For example, when entering the mean for a sample that includes 55 percent English learners,
enter "five five."
Enter the standard deviations for the measure, if the study reports them, or select Not Reported.
Select Yes or No for whether the analysis included a statistical adjustment for this
baseline measure.
If it did, select the method used.
If provided, enter the correlation between the outcome and the baseline measure.
Finally, select the appropriate effect size computation from options that include "Unadjusted
mean and standard deviation" and "t-statistic."
If you select t-statistic, the SRG will display a box in which you can enter the t-statistic.
We will select "Unadjusted mean and standard deviation" for our example.
Click Save and Continue to save the baseline measure.
Back on the main measures page, the SRG will display the result of the assessment of baseline
equivalence.
Here, we see that our measure of income does not demonstrate baseline equivalence and is
in the range that would require an adjustment in the analysis.
However, because the baseline measure is not required under the protocol and because the
study is a low-attrition RCT, the baseline measure will not influence the rating.
You may continue entering as many baseline measures as needed for this outcome measure.
In this section, we will enter information used for reporting the study's findings.
See the Reporting module, which is Module 7, for more information on how the WWC reports
and summarizes findings.
First, enter the unadjusted outcome means and adjusted outcome means for the analytic
sample.
Studies may report adjusted means if the authors adjusted for baseline differences in their
analysis.
If the study does not report one or both of these means, select Not Reported.
Then, enter the unadjusted standard deviation for each group, if provided.
Select Yes or No to answer the question "Does a negative result indicate a favorable outcome?"
In our example, reading achievement scores, the answer is No because a higher score is
a favorable outcome.
However, if our outcome measured dropout rates, for example, we would need to select Yes to
indicate that a lower dropout rate is a favorable outcome.
This ensures that the findings are reported correctly as evidence of favorable or unfavorable
effects of the intervention.
As discussed in the Reporting module, the WWC can calculate the study effect size from
several common analyses.
The SRG will adjust the information requested based on the effect size computation you select.
If there are multiple possible calculation methods, select the method appropriate for
the most rigorous method used in the study.
For example, if both adjusted and unadjusted means are reported, use the adjusted means
to calculate the effect size.
Let's say that our study used an ordinary least squares regression model for the analysis,
so we'll select OLS from the options.
Now, we need to enter the coefficient for the intervention status indicator variable
in the model and enter additional information the study provided about the effect size and
statistical significance.
Follow the guidance that appears when you click on the question mark next to the question
"Use in place of calculated value?"
The guidance will explain when the WWC will use author-reported values instead of values
calculated by the WWC.
In our example, we will select No for the author-reported effect size, indicating that
the authors did not calculate the effect size using the WWC formula for Hedges' g.
We will still enter the study-reported effect size into the SRG so that the data in the
SRG are as complete as possible.
We will also enter the study-reported p-value.
In this case, we will select Yes to use this value because the WWC generally accepts author-reported
p-values and statistical significance, unless there is a problem with the study calculations
or the values do not account for clustering in cluster-level assignment studies.
Next, we will follow the guidance that appears when you click on the question mark next to
"Is finding significant?"
We will leave the final row blank because the study provided a numerical p-value.
If the study only reported categorical p-values or p-values such as "less than .05," then
we would use this row to enter the statistical significance information.
Finally, enter any notes specific to this measure.
Use this space to document any concerns or questions about this measure, assumptions
you made while reviewing the measure, and additional information about the measure not
recorded in the SRG.
Remember, use complete sentences and reference pages where appropriate so that the review
will be accessible to others.
For example, although earlier we indicated that this study is a low-attrition RCT, suppose
instead we were missing data on the randomly assigned sample sizes.
We would enter, "This measure is missing information on the randomly assigned sample
sizes for the intervention and comparison conditions; therefore, we cannot calculate
attrition.
We will need to send an author query to confirm the randomly assigned sample sizes.
See pages 8 and 9 of the study for information provided on the sample sizes."
When you are done entering notes, select Save and Continue.
This will take you back to the main Measures page.
On this page, you will see a summary of the measure you just entered.
This includes the outcome domain, outcome measure name, the type of comparison, the
sample, and the follow-up period.
The page also shows the WWC rating for the measure.
In this case, because the measure is a randomized controlled trial with low attrition, it receives
the rating Meets WWC Group Design Standards Without Reservations.
If you disagree with the rating, first select Edit and check that all the information you
entered is correct.
If it is correct, note in the study notes why you disagree with the SRG-calculated rating
and what you think the correct rating should be.
To finish entering information about the outcome measure, you need to select whether the measure
is a Main or Supplemental finding.
This could depend on the time period of the outcome, whether the outcome was measured
for the full sample or a subsample, and whether the outcome measure is a subscale or a composite
score.
The Reporting module, which is Module 7, provides guidance on this decision.
The review protocol may also provide information about which outcome time point should be considered
the main finding.
In our case, we will designate this finding as the main finding.
You will also see a button to aggregate findings on this page.
Use this option if the study reports multiple findings, for example by grade level or subgroup,
but no overall finding, or if the overall finding is rated Does Not Meet WWC Group Design
Standards.
You can select the findings that should be aggregated and then click Save and Continue.
If you do not need to aggregate findings, then you can simply select Save and Continue
to move to the next review stage.
The Rating page allows you to review the information you entered about the study's outcome measures
and the final WWC design rating for the study.
Carefully review this page to ensure there are no mistakes in the information entered,
and that you agree with the ratings for each measure and for the overall study.
This is also an opportunity to edit the study notes to make sure that all questions, assumptions,
and concerns are documented.
The first section on the page summarizes the study information.
Here, you can check the study citation, the final WWC rating for the study, and a summary
of the findings that meet WWC design standards.
The remaining sections provide detailed information on each outcome measure entered in the SRG.
Check these to make sure information such as the sample sizes, means and standard deviations,
and effect sizes are entered correctly.
Note that some of these values are calculated by the SRG, so they may not match information
you entered.
When you have confirmed that all the information is correct, select Save and Continue to move
to the next stage.
If a study does not meet WWC design standards, the Rating stage will be the final stage.
After you have confirmed that all the information is correct, you will select Complete to finish
your review of the study.
If a study meets WWC design standards, like the study in our example, you have two additional
stages to complete: Context and Narrative.
As a reminder, these stages are required in the contract SRG, but they are optional in
the public SRG.
We will select Save and Continue to move to the Context stage.
On the Context page, you will enter information about the study's analytic sample.
Complete as much information as you can.
If the study reports on multiple samples, report information on the sample that corresponds
most closely with the analytic sample for the main finding.
The information entered here in the contract version feeds the sample characteristics reported
for the study on the WWC website and determines how interventions are classified for the Find
Research with Students Like Yours search feature.
First, we'll enter the total analytic sample size and select the sampling unit for the
study.
Next, we will select the grade levels for students in the analytic sample.
In our example, we'll say the intervention was given to 1st- and 2nd-grade students.
For race, let's say the authors reported information on the percentages of Asian, Black,
and White students.
Since we don't have information on the remaining percentages, we'll leave those blank.
The next category asks about sample members' ethnicity, so we'll enter the percentage
that authors report as Hispanic or Latino.
We'll continue through each category, entering as much information as the study provides.
For Class type, the options are Inclusion, meaning special education and general education
students are taught together, or a General Education classroom.
For this category and the remaining categories, you can select multiple options.
When you are done entering information, check the data you entered, then select Save and
Continue to move to the final stage.
On the Narrative page, you can enter additional information about the study.
Information entered in the contract version feeds the Study Details provided about the
study on the WWC website.
When you summarize study information on the narrative page, use your own words and do
not quote directly from the study.
Unlike the Context page, the fields on the Narrative page are text fields.
Be sure to use complete sentences and check your spelling and grammar when you complete
these fields.
Under each text box is a Notes box.
Include page numbers and other relevant information here about where in the study you found the
information for each section.
To provide an example of the type of information to enter into these fields, we will use the
text entered for the Glazerman (2006) study.
First, the SRG asks you to describe the setting of the study.
As the note says, include information about the study location and the types of classrooms,
courses, and schools involved in the study.
Next, you will need to describe the study design.
Earlier, you indicated whether the study design is an RCT or QED, but here you provide more
detail on the design.
Include information on how the intervention and comparison groups were formed, as well
as the recruitment and consent process.
The next box asks you to describe the study's sample sizes.
Again, you entered information earlier about the analytic sample sizes and, if the study
is an RCT, the randomized sample sizes.
This is an opportunity to summarize that information and provide additional detail, such as the
number of schools, classrooms, or teachers included in the sample, even if those are
not the unit of analysis.
In the next box, describe the sample characteristics.
This may include information you entered on the Context page, as well as additional relevant
information about the sample.
Check the review protocol for information about characteristics of interest.
As with the Context page, focus on describing the analytic sample in the study.
The next two boxes ask about the intervention condition and comparison condition, as implemented
in the study.
Include information on the content of the intervention and the comparison condition,
if available, as well as details such as the frequency with which the intervention or comparison
condition was implemented.
If the study does not provide much information about the comparison condition, include as
much information as possible or note that the comparison is assumed to be business as
usual.
Finally, describe any support for implementation in the final text box, if the study provides
this information.
This might include training or professional development provided to teachers, feedback
provided during the intervention, and any materials that participants received to implement
the intervention.
When you are done, review your responses to ensure they are complete and are free of spelling
or grammatical errors.
Then select Complete to finish your study review.
For more infomation >> The Essex Design Guide - Interview with Graham Thomas - Duration: 1:12. 
For more infomation >> The Essex Design Guide - Interview with Roy Warren - Duration: 1:09. 
For more infomation >> Explanimators | Episode 4 | Der animierte Guide zum Thema Machine Reading - Duration: 2:58. 
For more infomation >> The Essex Design Guide - Interview with Graham Cherry - Duration: 1:20. 
Không có nhận xét nào:
Đăng nhận xét