Menu Close

Conference Syntheses

Conference causal knowledge analytics – ECIS 2024

We propose an innovative approach to accelerate knowledge sharing, discovery, analysis, and synthesis in the AIS community. We will, where appropriate, assist scholars of accepted ECIS papers in coding their papers’ causal and process models as standardized graphs[1] for inclusion in a conference causal knowledge database. We will also add the coded models to an existing database of 238 MISQ curation papers.

Standardized digital representations of knowledge will enable scholars to query a conference database of IS knowledge at five levels (Figure 1). The analytical methods we have developed can compute metrics for a set of articles (e.g., dependent concept consensus) that are infeasible with current large language models. Analysaurus,[2] a web-based tool, enables scholars to query and analyze a graph knowledge database across all levels.

Figure 1: Five levels of analysis

Some benefits to ECIS attendees include:

  • Accelerated access to relevant conference papers (e.g., which papers investigate A → B)
  • Computing new types of metrics for conference proceedings (e.g., which causal model is most distinct from others)
  • Reporting a profile of a track in terms of concepts, relationships, and theories examined. (e.g., which conceptual relationships are frequently studied in a track)
  • Synthesizing across conference tracks. (e.g., which conference tracks align in terms of dependent concepts)
  • Comparing causal knowledge reported in ECIS proceedings and other IS literature. (e.g., which introduced concepts or relationships bridge the existing causal knowledge network)

Some benefits to the AIS community include:

  • This initial attempt to capture the core knowledge of an AIS conference is the first of any domain as far as we can assess. It will introduce a new set of metrics for measuring the general nature of a conference beyond submission counts.
  • Pioneering the development of tools for causal knowledge analytics.
  • Leading the social sciences in the digitization of knowledge.

ECIS pilot

  1. We propose contacting all authors whose accepted papers contain an explicit causal or process model.
  2. We will offer them an online workshop (we have already done about 10 of these), resources,[3] and individual help in coding their models.
  3. We will add all models to our existing database[4] with a tag identifying their source.
  4. We will provide a report to the program chairs based on our analysis of the papers. The report will provide a publication-level analysis, a within-track report, and a cross-track synthesis (Table 1). We do not propose to compute all the metrics listed in the table, but the database will be publicly available for others to augment our initial analysis. As the knowledge presented at other AIS conferences is captured, cross-conference analyses will be feasible.
  5. We will submit a report on the ECIS pilot to CAIS for consideration for publication.
  6. We propose the following tentative timeline for the ECIS conference digitization and synthesis.
    1. Week 1 April 2024: Online workshop for paper codification
    2. Week 2-3 April 2024: Authors code their papers
    3. Week 4 April – Week 3 May 2024: Preparing the conference report and submitting the report to program chairs for review.
    4. Week 4 May – Week 1 June 2024: Revising the conference report based on the feedback.

Table 1 Causal knowledge analytics for a conference

Level Analysis Purpose Measure
Paper Level of innovation Measuring the level of innovation of a causal model Conceptual isomorphism
Significance of knowledge contribution Measuring the extent to which concepts in a model are studied in the literature Effective size
Identifying models that are influential to causal mechanism Betweenness centrality
Identifying models that unite fragment knowledge Cutpoints
Track fit analysis Measuring the fit between a proceeding and its track theme LLMs
Track Concept semantic similarity analysis Computing concept similarity Natural language processing (NLP)
Concept position analysis Quantifying concepts’ network position Network science metrics
Reporting relationships between concepts Identifying concepts that tend to appear together in a model and reporting relationship frequency Association rule analysis
Assessing the level of knowledge fragmentation Evaluating the density of a knowledge network Network science metrics
Reporting theories Reporting theories and their frequency GQL
Conference Assessing the contribution of the conference proceedings Comparing the conference database with the existing literature database GQL, Network science metrics
Measuring the convergence among conference tracks Concept-track correspondence analysis Correspondence analysis
Theory-track correspondence analysis Correspondence analysis
Assessing causal consensus Measuring the level of agreement in the literature Information entropy

In summary, with the agreement and the support of the ECIS program committee, we invite you to participate in the first effort to enable causal knowledge analytics and provide a conference synthesis. It is a chance to show how digitization of knowledge can raise the productivity of IS scholarship and introduce a new set of metrics for evaluating a conference.

[1] Using Codasaurus, web-based software we have developed, we estimate it will take 15 minutes to code a model.


[3] Song, Y., Watson, R. T., & Zhao, X. (forthcoming). Digitized knowledge-based literature reviewing: A tutorial on coding causal and process models as graphs. Journal of Decision Systems.
Watson, R., Song, Y., Zhao, X., & Webster, J. (Forthcoming). Extending the Foresight of Phillip Ein-Dor: Causal Knowledge Analytics. Journal of the Association of Information Systems.