Conference causal knowledge analytics – ECIS 2024

We propose an innovative approach to accelerate knowledge sharing, discovery, analysis, and synthesis in the AIS community. We will, where appropriate, assist scholars of accepted ECIS papers in coding their papers’ causal and process models as standardized graphs[1] for inclusion in a conference causal knowledge database. We will also add the coded models to an existing database of 238 MISQ curation papers.

Standardized digital representations of knowledge will enable scholars to query a conference database of IS knowledge at five levels (Figure 1). The analytical methods we have developed can compute metrics for a set of articles (e.g., dependent concept consensus) that are infeasible with current large language models. Analysaurus,[2] a web-based tool, enables scholars to query and analyze a graph knowledge database across all levels.

Figure 1: Five levels of analysis

Some benefits to ECIS attendees include:

• Accelerated access to relevant conference papers (e.g., which papers investigate A → B)
• Computing new types of metrics for conference proceedings (e.g., which causal model is most distinct from others)
• Reporting a profile of a track in terms of concepts, relationships, and theories examined. (e.g., which conceptual relationships are frequently studied in a track)
• Synthesizing across conference tracks. (e.g., which conference tracks align in terms of dependent concepts)
• Comparing causal knowledge reported in ECIS proceedings and other IS literature. (e.g., which introduced concepts or relationships bridge the existing causal knowledge network)

Some benefits to the AIS community include:

• This initial attempt to capture the core knowledge of an AIS conference is the first of any domain as far as we can assess. It will introduce a new set of metrics for measuring the general nature of a conference beyond submission counts.
• Pioneering the development of tools for causal knowledge analytics.
• Leading the social sciences in the digitization of knowledge.

ECIS pilot

1. We propose contacting all authors whose accepted papers contain an explicit causal or process model.
2. We will offer them an online workshop (we have already done about 10 of these), resources,[3] and individual help in coding their models.
3. We will add all models to our existing database[4] with a tag identifying their source.
4. We will provide a report to the program chairs based on our analysis of the papers. The report will provide a publication-level analysis, a within-track report, and a cross-track synthesis (Table 1). We do not propose to compute all the metrics listed in the table, but the database will be publicly available for others to augment our initial analysis. As the knowledge presented at other AIS conferences is captured, cross-conference analyses will be feasible.
5. We will submit a report on the ECIS pilot to CAIS for consideration for publication.
6. We propose the following tentative timeline for the ECIS conference digitization and synthesis.
   1. Week 1 April 2024: Online workshop for paper codification
   2. Week 2-3 April 2024: Authors code their papers
   3. Week 4 April – Week 3 May 2024: Preparing the conference report and submitting the report to program chairs for review.
   4. Week 4 May – Week 1 June 2024: Revising the conference report based on the feedback.

Table 1 Causal knowledge analytics for a conference

Level	Analysis	Purpose	Measure
Paper	Level of innovation	Measuring the level of innovation of a causal model	Conceptual isomorphism
	Significance of knowledge contribution	Measuring the extent to which concepts in a model are studied in the literature	Effective size
		Identifying models that are influential to causal mechanism	Betweenness centrality
		Identifying models that unite fragment knowledge	Cutpoints
	Track fit analysis	Measuring the fit between a proceeding and its track theme	LLMs
Track	Concept semantic similarity analysis	Computing concept similarity	Natural language processing (NLP)
	Concept position analysis	Quantifying concepts’ network position	Network science metrics
	Reporting relationships between concepts	Identifying concepts that tend to appear together in a model and reporting relationship frequency	Association rule analysis
	Assessing the level of knowledge fragmentation	Evaluating the density of a knowledge network	Network science metrics
	Reporting theories	Reporting theories and their frequency	GQL
Conference	Assessing the contribution of the conference proceedings	Comparing the conference database with the existing literature database	GQL, Network science metrics
	Measuring the convergence among conference tracks	Concept-track correspondence analysis	Correspondence analysis
		Theory-track correspondence analysis	Correspondence analysis
	Assessing causal consensus	Measuring the level of agreement in the literature	Information entropy

In summary, with the agreement and the support of the ECIS program committee, we invite you to participate in the first effort to enable causal knowledge analytics and provide a conference synthesis. It is a chance to show how digitization of knowledge can raise the productivity of IS scholarship and introduce a new set of metrics for evaluating a conference.

[1] Using Codasaurus, web-based software we have developed, we estimate it will take 15 minutes to code a model.

[2] https://t-rex-graph.org/analysaurus/

[3] Song, Y., Watson, R. T., & Zhao, X. (forthcoming). Digitized knowledge-based literature reviewing: A tutorial on coding causal and process models as graphs. Journal of Decision Systems.
Watson, R., Song, Y., Zhao, X., & Webster, J. (Forthcoming). Extending the Foresight of Phillip Ein-Dor: Causal Knowledge Analytics. Journal of the Association of Information Systems.

[4] https://t-rex-graph.org/database/