Were the people drawn individually or in groups? (sampling design: strata and clusters)

In brief

Were the people drawn individually or whole schools and households? Such clusters and strata (groups such as regions) belong in the sampling design. Without it standard errors and intervals are far too small. With a survey weight Statable calculates standard errors, intervals and tests according to this design.

How it works in Statable

1. On the "Data" page choose the "Case weight" button and name the column with the weights.
2. Choose "Survey weight", then "No, I will state the sampling design".
3. Choose stratum, cluster and population from the columns. Everything is optional.

The information

• Stratum: a column that names the group from which people were drawn separately (for example the region or the type of school). Optional.
• Cluster: a column that names the unit that was drawn (for example the school or the household). Optional. With strata and clusters, a cluster number counts only within its stratum.
• Population: a number column with the size of the population per stratum (without clusters) or with the number of clusters per stratum (with clusters). Optional. It makes the standard error smaller when the sample is a large part of the population. If the value is smaller than the sample, Statable blocks with a sentence.
• Second stage: a column for the units within the clusters (for example the class within the school) and a number column with their number in the population. It is only offered when clusters and population are stated; without the number for the second stage it plays no role and is not used.

Step by step

1. On the "Data" page choose the "Case weight" button and name the column with the weights.
2. Choose "Survey weight". For the question whether the people were drawn individually, choose "No, I will state the sampling design".
3. Choose stratum, cluster and population from the columns. Everything is optional; without any information the people count as drawn individually.

The design is then shown below every result and in the report. It applies to all analyses with a survey weight: distribution, group comparison, cross table and regression.

Stratum with only one cluster

If a stratum has only one cluster, the spread within that stratum cannot be estimated. Statable then blocks and names the strata. You can choose one of four rules; it changes the standard error, the mean stays the same.

• Deviation from the overall mean (adjust): the stratum is calculated with its deviation from the overall mean. This is cautious and tends to give larger errors.
• Average of the other strata (average): the stratum varies as much as the other strata do on average.
• Contributes nothing (remove): the stratum contributes nothing to the spread.
• Selected with certainty (certainty): the cluster was selected with certainty, not at random; there is no spread from the selection there.

The chosen rule is named in the report. Example with a mean of 666.80: the standard error is 10.50 (adjust), 12.84 (average) and 10.48 (remove and certainty).

Design effect and effective sample size

The design effect (deff) says by which factor the design enlarges the variance compared with individually drawn people. With deff = 4 the standard errors are twice as large. The effective sample size is n divided by deff: that many individually drawn people would hold as much information. The size after Kish applies only to people drawn individually and is not shown once a design is stated.

Degrees of freedom

Tests and intervals use the degrees of freedom of the design: number of clusters minus number of strata. With few clusters that is few (df). Below 20 Statable points this out: the intervals are then wide and the approximation is weaker.

Example

A data set with 183 schools from 15 school districts (clusters). The mean of a test score is 644.17. With the design (clusters and population) the standard error is 23.54, the design effect 9.35, the effective sample size about 19.6 and df = 14. If the same data had been calculated as drawn individually, the standard error would be 7.82: three times too small. A wrong design therefore makes intervals far too narrow.

When Statable blocks

• The column for stratum or cluster has gaps. Rows never drop out silently; Statable names the number of rows.
• There are fewer than 2 clusters or more clusters than rows.
• The population is smaller than the sample.
• A stratum has only one cluster and no rule for it is chosen.

Important

Statable cannot check whether the design suits the survey. If the information is wrong, the intervals are too narrow or too wide. If in doubt, ask whoever collected the data, or choose "No or I do not know": the results then stay descriptive.

For specialists

R survey: svydesign with ids (clusters, second stage as ids = ~k + k2), strata, fpc and weights; with strata and clusters nest = TRUE. Variance by Taylor linearization, degrees of freedom degf (clusters minus strata), intervals with the t distribution. Stratum with one cluster: survey.lonely.psu with adjust, average, remove or certainty, set only for the call. Design effect deff = variance of the design divided by the variance under simple random sampling with the same n, without finite population correction; effective sample size n / deff. Checked on the example data apistrat, apiclus1 and apiclus2 against R survey.

Against SPSS Complex Samples (estimation "with replacement") means, totals, standard errors and regression coefficients are equal. Differences: SPSS ignores the population there, Statable uses it as soon as it is stated. In the regression, survey subtracts the number of coefficients minus 1 from the degrees of freedom (apistrat: 193 instead of 197 in SPSS).

Sources

Lumley, T. (2010). Complex Surveys: A Guide to Analysis Using R. Wiley. Lumley, T. (2004). Analysis of complex survey samples. Journal of Statistical Software, 9(8). Kish, L. (1965). Survey Sampling. Wiley.