Code for the DH project "Dhimmis & Muslims – Analysing Multireligious Spaces in the Medieval Muslim World"

Last update: Jul 01, 2022

Related tags

Overview

Damast

This repository contains code developed for the digital humanities project "Dhimmis & Muslims – Analysing Multireligious Spaces in the Medieval Muslim World". The project was funded by the VolkswagenFoundation within the scope of the "Mixed Methods" initiative. The project was a collaboration between the Institute for Medieval History II of the Goethe University in Frankfurt/Main, Germany, and the Institute for Visualization and Interactive Systems at the University of Stuttgart, and took place there from 2018 to 2021.

The objective of this joint project was to develop a novel visualization approach in order to gain new insights on the multi-religious landscapes of the Middle East under Muslim rule during the Middle Ages (7th to 14th century). In particular, information on multi-religious communities were researched and made available in a database accessible through interactive visualization as well as through a pilot web-based geo-temporal multi-view system to analyze and compare information from multiple sources. A publicly explorable version of the research will soon be available, and will be linked here. An export of the data collected in the project can be found in the data repository of the University of Stuttgart (DaRUS) (draft, not yet public).

Database

The historical data is collected in a relational PostgreSQL database. For the project, we have used PostgreSQL version 10. Since the project also deals with geographical data, we additionally use the PostGIS extension. A suitable database setup is to use the postgis/postgis:10-3.1 Docker image. An SQL script for creating the database schema is located in util/postgres/schema.sql, and in DaRUS. An overview of the interplay between tables of the database, and a general explanation, can be found in the docs/ directory.

Software

The server is programmed in Python using Flask. Functionalities are split up into a hierarchy of Flask blueprints; for example, there is a blueprints for the landing page, one for the visualization, and a nested hierarchy of blueprints for the REST API. The server provides multiple pages, as well as a HTTP interface for reading from and writing to the PostgreSQL database. The server is built and deployed as a Docker container that contains all necessary dependencies.

An overview and explanation of the different pages and functionalities is located in the docs/ directory. The web pages consist of HTML, CSS and JavaScript. The HTML content is in most cases served via Jinja2 templates that are processed by Flask. The JavaScript code is compiled from TypeScript source, and the CSS is compiled from SCSS.

Getting Started

Basic knowledge with build tools and Docker are required. The instructions below assume a Linux machine with the Bash shell are used.

Installing Dependencies

On the build system, Docker and NodeJS need to be installed. If the Makefile is used, the build-essentials package is required as well. In the root of the repository, run the following code to install the build dependencies:

$ npm install

Building the Frontend

To build all required files for the frontend (JavaScript, CSS, documentation), the Makefile can be used, or consulted for the appropriate commands:

$ make prod

Building the Docker Image

All frontend content and backend Flask code and contents are bundled within a Docker image. In that image, the required software dependencies are also installed in the correct versions. A few configuration options need to be baked into the Docker image on creation, which are dependent on the setup in which the Docker container will later run. Please refer to the deploy.sh shell script for details and examples, as well as to the section on running the server. The Dockerfile is constructed from parts from util/docker/, and then enriched with runtime information to ensure that certain steps are repeated when data changes. An exemplary creation of a Docker image (fictional values, please refer to deploy.sh before copying) could look as follows:

Dockerfile # build Dockerfile (warning: dummy parameters!) $ sudo docker build -t damast:latest \ --build-arg=USER_ID=50 \ --build-arg=GROUP_ID=50 \ --build-arg=DHIMMIS_ENVIRONMENT=PRODUCTION \ --build-arg=DHIMMIS_VERSION=v1.0.0 \ --build-arg=DHIMMIS_PORT=8000 \ .">

# calculate hash of server files to determine if the COPY instruction should be repeated
$ fs_hash=$(find dhimmis -type f \
    | xargs sha1sum \
    | awk '{print $1}' \
    | sha1sum - \
    | awk '{print $1}')

# assemble Dockerfile
$ cat util/docker/{base,prod}.in \
    | sed "s/@REBUILD_HASH@/$fs_hash/g" \
    > Dockerfile

# build Dockerfile (warning: dummy parameters!)
$ sudo docker build -t damast:latest \
    --build-arg=USER_ID=50 \
    --build-arg=GROUP_ID=50 \
    --build-arg=DHIMMIS_ENVIRONMENT=PRODUCTION \
    --build-arg=DHIMMIS_VERSION=v1.0.0 \
    --build-arg=DHIMMIS_PORT=8000 \
    .

The resulting Docker image can then be transferred to the host machine, for example, using docker save and docker load. Of course, the image can be built directly on the host machine as well.

Running the Server

The server infrastructure consists of three components:

The Flask server in its Docker container,
the PostgreSQL database, for example in the form of a postgis/postgis:10-3.1 Docker container, and
a reverse HTTP proxy on the host machine that handles traffic from the outside and SSL.

The util/ directory contains configuration templates for an NGINX reverse proxy, cron, the start script, the systemd configuration, and the user authentication file. The documentation also goes into more details about the setup. A directory on the host machine is mapped as a volume to the /data directory in the docker container. The /data directory contains runtime configuration files (users.db, reports.db, as well as log files). The main Docker container requires some additional runtime configuration, for example for the PostgreSQL password, which can be passed as environment variables to Docker using the --env and --env-file flags. The following configuration environment variables exist, although most have a sensible default:

Environment Variable	Default Value	Description
`DHIMMIS_ENVIRONMENT`		Server environment (`PRODUCTION`, `TESTING`, or `PYTEST`). This decides with which PostgreSQL database to connect (`ocn`, `testing`, and `pytest` (on Docker container) respectively. This is usually set via the Docker image.
`DHIMMIS_VERSION`		Software version. This is usually set via the Docker image.
`DHIMMIS_USER_FILE`	`/data/users.db`	Path to SQLite3 file with users, passwords, roles.
`DHIMMIS_REPORT_FILE`	`/data/reports.db`	File to which reports are stored during generation.
`DHIMMIS_SECRET_FILE`	`/dev/null`	File with JWT and app secret keys. These are randomly generated if not passed, but that is impractical for testing with hot reload (user sessions do not persist). For a production server, this should be empty.
`DHIMMIS_PROXYCOUNT`	`1`	How many reverse proxies the server is behind. This is necessary for proper HTTP redirection and cookie paths.
`DHIMMIS_PROXYPREFIX`	`/`	Reverse proxy prefix.
`FLASK_ACCESS_LOG`	`/data/access_log`	Path to `access_log` (for logging).
`FLASK_ERROR_LOG`	`/data/error_log`	Path to `error_log` (for logging).
`DHIMMIS_PORT`	`8000`	Port at which `gunicorn` serves the content. Note: This is set via the Dockerfile, and also only used in the Dockerfile.
`PGHOST`	`localhost`	PostgreSQL hostname.
`PGPASSWORD`		PostgreSQL password. This is important to set and depends on how the database is set up.
`PGPORT`	`5432`	PostgreSQL port
`PGUSER`	`api`	PostgreSQL user

Comments

What data to include in Life Version?

Initially opened by gs108488 in [email protected]. History:

gs108488 Dec 15, 2021:

Similar to issue #172, we need to define which data is included in the Life Version?

help wanted question

opened by mfranke93 40
Check texts in Help/Info menus

Initially opened by gs108488 in [email protected]. History:

gs108488 Dec 16, 2021:

All texts in info boxes when clicking on ? symbols in visualization need to be checked, especially in respect to publishing the life version.

gs108488 Dec 16, 2021:

I put this on halt until the state of the visualization for the life version is really final.

documentation enhancement

opened by mfranke93 37
Change text and appearence for start page of Life Version

Initially opened by gs108488 in [email protected]. Relevant history:

gs108488 Dec 16, 2021:

For the Life Version, the start page needs a complete make over. Apart from "English only" (see ~~#210~~), the start page will need different sections. For the design, the styles inherited from the overall base.htm need to be considered.

enhancement

opened by mfranke93 31
Explanation of "Aggregation of religious groups"

In the info text of the map, I find the following sentences:

If in all shown map glyphs no more than four nodes of a lower part in the religion hierarchy would be present, the data is aggregated on that lower level. For example, if the map would only show two glyphs; one with the Latin Church, the Coptic Church, and the Georgian Orthodox Church; and the other with the Latin Church and the Rabbanites; each of these religious denominations could be represented by an individual circle. This will only happen if it is possible in all glyphs, as doing otherwise would skew the perceived variety of religions.

I hardly understand this. Is there something wrong, e.g. with the first phrase in the <em> tag?
help wanted question

opened by tutebatti 23
Include section "How to Cite" in Report
Where would I suggest what the section needs to look like? This is not as easy compared to changing the html of the info texts, is it?

What is more, I am not sure who the "authors" of a report are when it comes to citation. Probably, instead of some standard bibliographical data, it should be something like:

We suggest to cite this report in the following way: "Report ###UUID### created by ###YOUR NAME### based on the visualization and data of Weltecke, Dorothea et al. "Dhimmis and Muslims. A Tool [...]" accessible via ###URI###."

enhancement
opened by tutebatti 20
Problems with cookie dialogue in Firefox?

Yesterday, someone from our team in Berlin tried visiting the public instance at damast.geschichte.hu-berlin.de with Firefox (on Windows) and the "Accept" button in the cookie dialogue did not respond. On my PC, with Firefox (on Linux) I got a grayed-out start page but not dialogue at all.

Happy about any recommendations to track these bugs.
bug

opened by tutebatti 19
Example when explaining regular expressions for "Place search"

In the current example in the info text for the search of places, one reads:

The search field supports JavaScript-style regular expressions. For example, to search for locations with an Arabic definite article, the query \ba([tdrzsṣḍṭẓln]|[tds]h)- can be used.

If I understand correctly from the list of places, we do not use the DMG notation for Arabic articles (cf. https://de.wikipedia.org/wiki/DIN_31635). That example makes little sense, then. Any better suggestions. @rpbarczok, you probably no the data itself better than @mfranke93?
help wanted discussion

opened by tutebatti 16
"NULL" vs. empty in column "comment" of Place table

I'm currently looking through the Place table, especially to check for major inconsistencies in the column comment, which is part of the Place URI page. On the one hand, there were 3 or for rows with linebreaks. On the other hand, some rows have nothing in the column comment, some have NULL. Is that normal?

Note: I am editing a csv downloaded via pgAdmin in LibreOffice Calc.
help wanted question

opened by tutebatti 15
Listing pieces of evidences in place URI pages?

Connected to #144 and #146, a reason to generate reports is that it is the only way to have access to the detailed pieces of evidences. The visitor seemingly wanted to create reports to read about the different pieces of evidence of a specific place. What would you say, @rpbarczok?

opened by tutebatti 12
Access to place URI page from map

Another comment by the same visitor giving feedback reported in #144:

It would be great to access the place URI page directly from the map.

Right now, the "mechanics" are either hovering displaying a tooltip. One can probably not use that as a link to the place URI page as it vanishes when moving the mouse. Clicking on the place (or the glyph) brushs and links, which then allows to go to the place URI page from the location list. So there is probably no easy solution. Maybe one could open the same popup as with the hovering with a right-click and make the tool tip persistent? Just an idea.
enhancement

opened by tutebatti 12
Changes on the automatically generated report

I looked through the record and I like to to suggest the following improvements:

1.) Query report: The first three filters of the query report “evidence must be visible”, “place must be visible” and “place type must be visible” is to my knowledge not part of the filter possibilities of the visualization. Someone has to know the data structure quite well to understand these statements. Therefore, I believe the users will be confused by them. Additionally, to my knowledge, no data in the DhiMu project is hidden. Therefore, I would like these statements to be removed if there are no other objections.

2.) Evidence a) I would rather have the evidences sorted by a) city name, b) religion name, c) start of the time span b) In the English lexicography there is not plural form of Evidence. An English speaker would rather use the term “Pieces of Evidence”. I think we have to change that. c) The evidence fragment of the report distinguishes between hidden and visible data. Is there a case when hidden data is shown in the report? To my knowledge hidden data is not part of the publicly available filtering system? In any case: No hidden evidences, places or region should be part of the DhiMu-data, so it does not need to be included in the report. d) I would like to move the footnote with the evidence comment to the end of the sentence. e) We should use “…religious group…” instead of Religion f) If I understand the source code correctly, a place with the category “unknown” is displayed as “place”. Maybe it would be better to add an “undefined” to it: “undefined place”. What do you think? g) I would rather have a colon between the short title of the source and the content of the source_instance.source_page cell. h) I would like to move the source comment into a footnote at the end of the sentence.

3.) Places a) According the place type "unknown", I suggest to add "undefined" to "place" as above. b) I would like to have the list items in the „Linked to“ section in the following way:

<place.name> is linked to: Digital Atlas of the Roman Empire (DARE): http://imperium.ahlfeldt.se/places/22329

i.e. without the short title and the ID at the beginning.

@tutebatti, @mfranke93: what do you think?
enhancement discussion

opened by rpbarczok 12

Releases(v1.1.5+history)

v1.1.5+history(Dec 6, 2022)

Source code(tar.gz)
Source code(zip)

Owner

University of Stuttgart Visualization Research Center

GitHub Repository

Python scripts aim to use a Random Forest machine learning algorithm to predict the water affinity of Metal-Organic Frameworks

The following Python scripts aim to use a Random Forest machine learning algorithm to predict the water affinity of Metal-Organic Frameworks (MOFs). The training set is extracted from the Cambridge S

1 Jan 09, 2022

Fit models to your data in Python with Sherpa.

Table of Contents Sherpa License How To Install Sherpa Using Anaconda Using pip Building from source History Release History Sherpa Sherpa is a modeli

134 Jan 07, 2023

Amundsen is a metadata driven application for improving the productivity of data analysts, data scientists and engineers when interacting with data.

3.7k Jan 03, 2023

A 2-dimensional physics engine written in Cairo

38 Nov 16, 2022

A columnar data container that can be compressed.

Unmaintained Package Notice Unfortunately, and due to lack of resources, the Blosc Development Team is unable to maintain this package anymore. During

944 Dec 09, 2022

LynxKite: a complete graph data science platform for very large graphs and other datasets.

LynxKite is a complete graph data science platform for very large graphs and other datasets. It seamlessly combines the benefits of a friendly graphical interface and a powerful Python API.

124 Dec 14, 2022

.npy, .npz, .mtx converter.

npy-converter Matrix Data Converter. Expand matrix for multi-thread, multi-process Divid matrix for multi-thread, multi-process Support: .mtx, .npy, .

1 Feb 07, 2022

MS in Data Science capstone project. Studying attacks on autonomous vehicles.

Surveying Attack Models for CAVs Guide to Installing CARLA and Collecting Data Our project focuses on surveying attack models for Connveced Autonomous

1 Dec 09, 2021

Port of dplyr and other related R packages in python, using pipda.

Unlike other similar packages in python that just mimic the piping syntax, datar follows the API designs from the original packages as much as possible, and is tested thoroughly with the cases from t

179 Dec 21, 2022

WAL enables programmable waveform analysis.

This repro introcudes the Waveform Analysis Language (WAL). The initial paper on WAL will appear at ASPDAC'22 and can be downloaded here: https://www.

40 Dec 13, 2022

Stitch together Nanopore tiled amplicon data without polishing a reference

Stitch together Nanopore tiled amplicon data using a reference guided approach Tiled amplicon data, like those produced from primers designed with pri

14 Aug 30, 2022

Data exploration done quick.

Pandas Tab Implementation of Stata's tabulate command in Pandas for extremely easy to type one-way and two-way tabulations. Support: Python 3.7 and 3.

20 Aug 27, 2022

BIGDATA SIMULATION ONE PIECE WORLD CENSUS

ONE PIECE is a Japanese manga of great international success. The story turns inhabited in a fictional world, tells the adventures of a young man whose body gained rubber properties after accidentall

3 Jun 30, 2022

Tools for working with MARC data in Catalogue Bridge.

catbridge_tools Tools for working with MARC data in Catalogue Bridge. Borrows heavily from PyMarc

1 Nov 11, 2021

Codes for the collection and predictive processing of bitcoin from the API of coinmarketcap

5 Apr 26, 2022

GWpy is a collaboration-driven Python package providing tools for studying data from ground-based gravitational-wave detectors

GWpy is a collaboration-driven Python package providing tools for studying data from ground-based gravitational-wave detectors. GWpy provides a user-f

342 Jan 07, 2023