Multiple working hypotheses
Multiple working hypotheses: Required reading
Chamberlain, Thomas C. 1890, The method of multiple working hypotheses: Science, v. 15, p. 92–96. Find this article online  clean version in pdf Commentary
Platt, John R. 1964. Strong Inference: Certain systematic methods of scientific thinking may produce much more rapid progress than others. Science 46(3642):347-353. http://pages.cs.wisc.edu/~markhill/science64_strong_inference.pdf
The Basics of Modeling (and Multiple working methods)
Select some of your core problem areas for which you have or can get data. Explore the set of possible models that can help you test explanations. Formulate a series of possible hypotheses that can be tested within one or more modeling approach.
Test each model against the null hypothesis of statistical independence - Measure departures from the null hypotheses. Study the patterns of these departures.
Test each model against the other -- Likelihood Ratios for comparative fit
Get a Maximal Likelihood Estimation for each model, using the K-S or Kolmogorov-Smirnov test, for which a probability closer to one is a better fit to bootstrap samples of the same size as your data sample, but coming from variates generated by model.
Reconstruct a complex model by comparisons of its outcome variables ... to the empirical data you have available.
If you use simulation, measure the outcome variables and compare the simulated data to those of the empirical data.
Meehl, Paul. 1967 Theory testing in Psychology and Physics: A Methodological Paradox. Philosophy of Science 34(2)103-115. reprinted Abstract: Because physical theories typically predict numerical values, an improvement in experimental precision reduces the tolerance range and hence increases corroborability. In most psychological research, improved power of a statistical design leads to a prior probability approaching '/z of finding a significant difference in the theoretically predicted direction. Hence the corroboration yielded by "success" is very weak, and becomes weaker with increased precision. "Statistical significance" plays a logical role in psychology precisely the reverse of its role in physics. (DRW: this problem is made vastly worse for the social sciences in that nonindependence of cases increases the spurious likelihood of finding differences, including different from the null hypotheses, and vastly decreases the ability of statistical tests to detect replication. Bottom line: better to deal with point data where possible, with the problem of nonindependence, network effects, and processes through time.)
Look at how each of your empirical variables, discrete or continuous, are distributed.
useful but advanced reading: 2007 review article, "Power-law distributions in empirical data," by Aaron Clauset, Cosma Rohilla Shalizi, and M. E. J. Newman. (has full [R and Matlab software and documentation]). See discussion of another distribution, at Tsallis q distribution project
___Higher order and Nonlinear Interactions
Three-way interactions. If there are 3-way interactions among your variables that are significant departures from randomness conditioned upon bivariate distributions,
If bivariate interactions are nonlinear (third and fourth moments depart from randomness)
there is little point using the standard GENERAL LINEAR MODEL (bivariate only, linearly related). This is the most common analytical mistake in the social sciences.
If your data are binary (0/1 possibly with missing data) you can do 3-way tests for higher order interaction, followed by [http://eclectic.ss.uci.edu/~drwhite/entail/sea.html Statistical Entailment Analysis, both are DOS programs, but Carter Butts
Illustrative types of models and logics of theory
- General linear model (rarely applies): e.g., Weller, Susan C. (2007) Cultural consensus theory: Applications and frequently asked questions. Field Methods 19: 339-368.
- Probabilistic gemerative model, e.g., Social-circles network model White, Tsallis
- Probabilistic fitted model, e.g., as above - Clauset et al
- Implicational logics: e.g., 1977 Douglas R. White, Michael L. Burton, and Lilyan A. Brudner, Entailment Theory and Method: A Cross-Cultural Analysis of the Sexual Division of Labor. Cross Cultural Research 12:1-24.
- Correlational radex theory - key variable that explains many: e.g. Radex theory of complex interactions
- Causal autocorrelation tests: e.g., 1988. Causes of Polygyny: Ecology, Economy, Kinship, and Warfare. Douglas R. White; Michael L. Burton. American Anthropologist, 90(4):871-887. Replicated by Malcolm Dow.
- Time-lagged correlation tests for dynamics: White, Tambayong, Keyzar - 2008 Oscillatory dynamics of city-size distributions in world historical systems. (drw, L. Tambayong, and N. Kejžar). In, G. Modelski, T. Devezas and W. Thompson, eds. Globalization as Evolutionary Process: Modeling, Simulating, and Forecasting Global Change. pp. 190-225. London: Routledge. http://intersci.ss.uci.edu/wiki/pw/ModelskiCh9WTK.pdf
- Network dependencies (e.g., micro/macro): White and Johansen