Sampling

There are a variety of probability sampling strategies. Frequently used probability sampling strategies are simple, systematic, stratified, and cluster sampling. Simple random sampling may be the best known sampling strategy. Systematic random sampling uses a list of population elements. If elements can be assumed to be randomly listed, then a starting point is randomly identified, and elements are selected using a sampling interval. These intervals are calculated by dividing the desired sample size by the number of elements in the sampling frame. For example, to select a sample size of 50 from a sampling frame with 100 elements, pick a random starting point and select every second element until 50 elements are selected. Stratified sampling uses groups to achieve representativeness, or to ensure that a certain number of elements from each group are selected. In a stratified sample, the sampling frame is divided into nonoverlapping groups or strata (e.g., age groups, gender). Then a random sample is taken from each stratum. This technique, for example, can be used to study a small subgroup of a population that could be excluded in a simple random sample. Cluster sampling enables random sampling from either a very large population or one that is geographically diverse. An important objective of cluster sampling is to reduce costs by increasing sampling efficiency. A problem with cluster sampling is that, although every cluster has the same chance of being selected, elements within large clusters have a greatly reduced chance of being selected in the final sample. Using the probability proportionate to size (PPS) technique corrects this error. PPS takes into account the difference in cluster size and adjusts the chance that clusters will be selected. That is, PPS increases the odds that elements in larger clusters will be selected [1]. All of these methods are available with the SURVEYSELECT procedure.

The SURVEYSELECT procedure provides a variety of methods for selecting probability-based random samples. The procedure can select a simple random sample or can sample according to a complex multistage sample design that includes stratification, clustering, and unequal probabilities of selection.

Let's use an imaginary dataset to demonstrate each sampling method. For example let's assume we have dataset that lists number of accidents per each day of a week and we would like to get a sample out of it. A few observations from the dataset looks like this (you can download the full dataset from here.)

Here's a summary of the original dataset:

Simple Random Sampling

The method of simple random sampling without replacement (METHOD=SRS) selects units with equal probability and without replacement and also the default sampling method in SAS. Each possible sample of different units out of has the same probability of being selected. After a unit is selected it cannot be selected again. Here's how we do it in SAS:

Here's a summary of our sample:

The problem with our sample is the distribution of days; clearly Saturday and Sunday are overrepresented whereas Monday and Wednesday are underrepresented. To fix this we have to specify a strata preceded by sorting the data by specified strata (this is one of the things that I still don't get - why this sorting cannot be done by the process itself but has to be separately conducted):

Note we specified the sample size as 14 for each strata so that we have a sample size of 14x7=98.

Strata option worked well in our example because there are equal number of observations for each day of the week but what if we have a sample with unequal distribution of days of week and we represent this proportion in our sample. Assume for example that our original data looked like this instead:

To get an accurate sample representing the original distribution of days of the week, we can use METHOD=SEQ, sequential random sampling which would take the original distribution into account without a need to specify strata at all. Here's how we do it:

If we instead want to create a sample with replacement instead, we should use METHOD=URS (unrestricted random sampling). With replacement in place of course, there is always a chance of selecting the same observation more than once. By default SAS removes duplicate observations from the final dataset. Let's demonstrate this with an example:

As can be seen, we have 13 observations for Thursday instead of 14 we specified. In order to keep the duplicate instead we add OUTHITS option to our code.

Example: Titanic Passengers

Description: TThe sinking of the RMS Titanic is one of the most infamous shipwrecks in history. On April 15, 1912, during her maiden voyage, the Titanic sank after colliding with an iceberg, killing 1502 out of 2224 passengers and crew. This data table describes the survival status of 1309 of the 1324 individual passengers on the Titanic. Information on the 899 crew members is not included

name: Passenger name.
survived: Yes or No.
passenger_class: 1, 2, or 3 corresponding to 1st, 2nd, or 3rd class.
sex: Passenger sex.
age: Passenger age.
siblings_and_spouses: The number of siblings and spouses aboard.
parents_and_children: The number of parents and children aboard.
fare: The passenger fare.
port: Port of embarkment (C = Cherbourg; Q = Queenstown; S = Southampton).
home_dest: The home or final intended destination of the passenger.

Source: Building Better Models with JMP Pro
Download the data from here

Task: Create a data set with %70 train and 30% test, derive a logistic regression model to find factors effecting survival chance with train data and test it with test data.

Let's first create our test and train dataset with SURVEYSELECT procedure:

RATE defines the percent of data to be selected - 30% in our case.
OUTALL tells SAS to output not just the selected but also unselected data in the output. To separate the selected from unselected, SAS will create a column called "selected" with value 1 for selected and 0 for unselected.
METHOD=SRS specifies the selection method = sampling with replacement in this case. Other option is URS.
SEED=1234 is simply a seed number for generating random selection. This can be any number. If you use the same number you will get the same result - it's good for reproducibility.

Now we have a new dataset with a "selected" column and we would like to create two datasets with names titanic_train and titanic_test. To do this we use DATA procedure:

Next we can run our first logistic regression model with train data:

Variables parents_and_children and fare are relevant so we will remove them from our model. Also we will add a new option to PROC LOGISTIC statement: OUTMODEL=titanic. This will save the model (formula) for making estimations on the test data set.

Let's now see how good our model is with our test data set.

Our error rate is 20%, not bad. Another measure I typically look into is Brier Score. Brier Score is the mean squared difference between the predicted probability and the actual outcome. The lower the Brier score is, the better our model is. A Brier score of zero means perfect fit and one means worst fit possible. For our case it's 0.15, not so bad - not amazing either. If we had randomly estimated survival, our Brier score would have been around 0.25, so 0.15 is not too bad.

When saving scores, SAS puts two columns in the output: f_survived and i_survived for each observation. f_survived is the value of survived in the titanic_test dataset and i_survived is the estimated value of the survival based on the logistic model. Therefore the more agreeement between these columns, the better our model is. We can utilize PROC FREQ to see how good our estimates are:

And here is our output:

p-value for McNemar test is 0.2113. Higher p-values mean better agreement. We can use this measure to compare different models. Agree plot shows our fit visually. Note that this measure would only be useful for relatively balanced designs.

Preventing Bias in Sampling

It is important to create samples that represent original data accurately otherwise we would have a biased sample. For example, take a look at the carddefault data which shows the credit card defaults with other socio-economic variables. A predictive model would try to estimate the defaulting. Let's take a look at the default and student ratios in the dataset via PROC FREQ:

Now let's create a training sample from our dataset via PROC SURVEYSELECT and see the distribution via PROC FREQ:

As can be seen from above default=Yes is oversampled and student=Yes is undersampled. To remedy this we have to use STRATA statement with PROC SURVEYSELECT. Note that sorting the data is required before invoking STRATA.

1. Determining Sample Size: Balancing Power, Precision, and Practicality, P. Dattalo