Did the measure work? Reading an interrupted time series

In brief

Did a measure work? The interrupted time series asks whether the series changes in level from a chosen point in time (step) or deviates only at one point in time (pulse). For this Statable estimates an ARIMA model, only on a click.

How it works in Statable

1. In the card "How does a value develop over time?" open the section "Did the measure work?".
2. "The measure applies from": choose the point in time fixed beforehand (required).
3. "Kind of change": step, step and pulse, or pulse only.
4. "Calculate the effect". Read the checklist first, then the numbers.

When this analysis fits

• There is a series with equal intervals (at least 20 values present, at most 10% gaps, not all values equal) and a point in time from which something is different: a new rule, a new offer, a price change.
• The point in time was fixed beforehand. It is required and has no default.
• There are enough values before and after the point in time: with a season almost two periods (2 m − 1 values), otherwise at least 8. If the point in time is at the edge of the series, the step would be the same everywhere; Statable then does not calculate.

The choices

• Type: step (0 before, 1 from the point in time on; default), step and pulse (the pulse is 1 only in the point in time itself) or pulse only. A step asks for a lasting change of the level, a pulse for a one-time swing.
• Scale: the log scale works only with values above 0. The effect then counts as a percentage of the level (exp(b) − 1), and the forecast is a median.
• Model: p, q, P and Q from 0 to 5, d up to 2, D up to 1; the seasonal part only with a period of 2 to 24. A constant exists only with d + D = 0 and only if it is chosen.
• Suggestion: "Suggest a model from the time before" chooses the model only from the values before the measure, so that the effect does not influence the choice. The suggestion calculates nothing; only "Adopt suggestion" makes it the choice. With gaps in the series there is no suggestion; the model can then only be chosen yourself.
• Forecast: number of points in time after the last value. The forecast assumes that the step remains and the pulse does not return.

Reading the result

• The checklist comes before the numbers. The warnings in it always apply (see below).
• The sentence says "went along with", never "caused": the measure went along with a decrease or increase of the level by an amount.
• For the step and the pulse the table shows the estimate (b), the standard error, the 95% interval (after Wald) and p. z = b/standard error applies; p comes from the normal distribution. On the log scale the change in percent is shown as well.
• The chart shows the series with a line at the point in time. Below are the goodness of fit, the forecast (the step remains) and the command in R.
• Ljung-Box checks whether the errors of the model are still related. The degrees of freedom are lags minus estimated ARMA parameters; step and pulse do not count.

Trap: point in time and cause

The model only measures whether the series changes at the chosen point in time. It does not show whether the measure is the cause. Everything that happened at the same time is part of the effect.

All warnings

• Cause open: everything that happened at the same time is part of the effect.
• Without a comparison group this stays open. A similar series without the measure would be stronger evidence.
• Fix the point in time beforehand. Anyone who looks for it in the data and then tests gets p-values that are too small.
• If the measure is phased in, acts with a delay or was expected, a step at the exact point in time does not fit.
• Fewer than 50 values: this is a usual lower limit for ARIMA (Box and Jenkins); the choice of model is uncertain.
• Ljung-Box with p < .05: the model does not capture the whole pattern, the standard errors are too small, and the interval and p-value of the effect are too optimistic.
• A coefficient at the edge (for example MA 1 at −1): the model is almost unstable, the standard errors and AICc are not reliable. Choose another model.
• If the standard error is missing for some of the parameters, the interval and p-value are missing there. Choose a simpler model.
• Log scale: the effect counts as a percentage of the level, the forecast is a median, not a mean.

Example

Monthly number of drivers killed or seriously injured in Great Britain, seat belt law from February 1983: with ARIMA(1,0,0)(0,1,1)[12] the step is −323.2 (standard error 73.5), a decrease of about 323 per month. But the Ljung-Box test is conspicuous (Q(24) = 78.06, p < .001): the model does not capture the pattern fully, the standard errors are probably too small, and the interval is too optimistic. A second example: for the Nile level with a step in year 29 the MA coefficient is −0.9999, that is, at the edge; Statable reports the model as not reliable.

Reporting the result

With the seat belt law from February 1983 a decrease of the level by 323 drivers per month went along (ARIMA(1,0,0)(0,1,1)[12], step b = −323.21, SE = 73.53, z = −4.40, p < .001, 95% CI [−467.3, −179.1]). Ljung-Box Q(24) = 78.06, p < .001: the residuals are still related, so the interval is too optimistic. This model does not show whether the law is the cause.

Differences from SPSS

• The standard errors differ by 5 to 10%. Statable uses the R function arima (maximum likelihood); SPSS uses a different method and a different variance. Example: step −323.21 (SE 73.53) in Statable against −323.20 (SE 71.16) in SPSS; the error variance is 19200.3 against 19525.9. The estimates agree; p and interval can differ.
• The MA coefficients are written the way R does it (Box and Jenkins); SPSS shows them with the opposite sign.

Sources

Box, G. E. P., & Tiao, G. C. (1975). Intervention analysis with applications to economic and environmental problems. Journal of the American Statistical Association, 70, 70-79.