Architecture

This page describes how the SDK is built: the rules every module follows, how the modules depend on each other, what a dataset is, and how a vote and a level of exposure are read. It is written for whoever reads the code, reviews it, or writes a source.

Six rules

  1. A load is frozen. A Dataset is everything a source gave at one instant (as_of), at one level of exposure, as one graph of frozen objects. No navigation reads the network. Loading again gives another dataset.

  2. The objects are frozen dataclasses, equal by identity in the product. Two objects are equal when they have the same class, the same id and the same dataset. A link is an id; the linked object is read through a property that asks the indexes of the dataset.

  3. The core imports neither the engine nor NumPy. import civiqhub loads the standard library only, and a test proves it. The analyses will live in the engine civiqhub-selene; the SDK holds no formula.

  4. Three states per vote. An Actant voted on a Revision, abstained (the Revision was shown, and no vote came), or never saw it. When the data do not record what was shown, the state is unknown. An absence is never a zero nor a neutral answer.

  5. Exposure. At the aggregated level, a dataset holds neither individual votes nor Actants; asking for them raises ExposureError, which says how to get the pseudonymised level. No name, e-mail address or user id of a person enters an object: an Actant is known by its pseudonym only. The token appears in no repr, file or source.

  6. Every analysis will say how to find it again. A result of analysis will carry the result of the engine and what it takes to compute it again: the Project, the instant, the fingerprint of the dataset, the method, its version, its settings, the seed and the fingerprint of the computation. This rule waits for the bridge to the engine. The word “citation” is kept for the citation of a Proposal in an article.

Modules

        flowchart TD
  client["client<br>Client, connect, load"] --> sources["sources<br>Source, MemorySource, FileSource, ApiSource"]
  client --> dataset["dataset<br>Dataset"]
  synthetic["synthetic<br>make_rows, make_truth, make_project"] --> client
  sources --> dataset
  dataset --> entities["entities<br>Entity, Group, Project, Article, ArticleVersion,<br>Paragraph, Placement, Citation, Proposal,<br>Amendment, Revision, Argument, Actant, Member,<br>Agent, Vote"]
  entities --> values["values<br>Exposure, Role, ActantKind, AmendmentStatus,<br>Position, VoteState, VoteCounts, Participation"]
  entities --> listing["listing<br>Listing"]
  values --> errors["errors<br>CiviqHubError and its seven errors"]
  listing --> errors
    

Module

Holds

Imports

civiqhub.errors

CiviqHubError and its seven subclasses; each carries a hint

standard library

civiqhub.values

The closed choices of the product, VoteCounts and Participation

standard library

civiqhub.listing

Listing, the frozen list of every collection

standard library

civiqhub.entities

Entity and the objects of the product

standard library

civiqhub.dataset

Dataset, the format of the rows and its checks

standard library

civiqhub.sources

Source and the three sources of the package

standard library (urllib for the API)

civiqhub.client

Client, connect(), load()

standard library

civiqhub.synthetic

make_rows(), make_truth(), make_project()

standard library (random, seeded)

Not written yet: the bridge to the engine, civiqhub.analysis.

The dataset

A source gives rows: a dictionary that JSON can write, in the format civiqhub.dataset.FORMAT, with one list of rows per section (groups, projects, articles, article_versions, paragraphs, placements, citations, proposals, revisions, vote_counts, amendments, arguments, and at the pseudonymised level actants, votes, impressions). A Dataset checks these rows once, when it is built, and gives a graph of frozen objects.

        flowchart LR
  source["a Source<br>API, file, memory, yours"] -- "fetch(exposure=...)" --> rows["rows<br>civiqhub.dataset/0"]
  rows --> dataset["Dataset<br>checked once, then frozen"]
  dataset --> objects["Group, Project, Proposal,<br>Revision, Vote, ..."]
  dataset -- "save(path)" --> file["JSON file"]
  file -- "civiqhub.load(path)" --> dataset2["the same rows,<br>the same fingerprint"]
    

Why frozen:

  • One instant. Every object of a dataset describes the same moment, as_of. Mixing objects read at two moments would make counts that never existed together.

  • No hidden reading. Navigating from a Proposal to its Revisions, its Amendments or its votes reads indexes built once. It never calls the network, so it never fails halfway and never costs a request.

  • Reproducible. fingerprint() identifies the rows: the same rows give the same fingerprint on every system. save() writes them to a file that reopens with the same fingerprint, so that a result can be computed again on the same data.

A rule that the rows break raises ValidationError, whose message names the row and the rule. A value that the model deprecated does not break a rule: an old dataset that holds the status DRAFT, the types CREATE and REMOVE, the pending changes, the kind SYNC or the traces of a fork loads, and deprecated marks each of these values. The SDK never produces one. To see newer data, load again: Client.refresh forgets what a client kept, and the objects already given stay valid in their own dataset.

Three states of a vote

Actant.vote_state_on gives a VoteState:

State

When

voted

The Actant voted on the Revision

abstained

The Revision was shown to the Actant, and no vote came

not_shown

The Revision was never shown to the Actant

unknown

No vote, and the data do not record what was shown

The data record what was shown when the rows hold the section impressions: Dataset.has_impressions says so. A vote always counts as shown.

Levels of exposure

Exposure has two levels:

Level

Holds

Who can read it

aggregated

The vote counts of each Revision and the participation of the Project; no Actant, no individual vote

Any token of the Project; the default of connect()

pseudonymised

One row per vote, each Actant known by a pseudonym; the authors of Proposals and Arguments

A token of a Project whose administrator enabled it

Data never gain detail: rows held at the aggregated level cannot be served at the pseudonymised level. Pseudonymised rows asked for at the aggregated level lose their individual sections and their authors, and keep the vote counts. civiqhub.sources.at_exposure() does both, for every source.

Sources

A source is the port through which data enter the SDK. It has one duty: Source.fetch gives rows in the format of a dataset, at the level asked for. Source.describe says where they come from, never with a secret; the dataset shows it as its source.

Source

Reads

ApiSource

The CiviqHub API, with a personal access token. Provisional: the API is not published, and its paths are assumptions kept in ApiSource.PATHS

FileSource

A file written by save()

MemorySource

Rows held in memory: tests, simulations, the data of another system

A source of your own derives from Source and works with every client, without a change in the package:

from typing import Any

import civiqhub
from civiqhub.sources import at_exposure


class SpreadsheetSource(civiqhub.Source):
    name = "spreadsheet"

    def __init__(self, rows: dict[str, Any]) -> None:
        self.rows = rows

    def fetch(self, container=None, *, exposure):
        return at_exposure(self.rows, exposure)

    def describe(self) -> str:
        return "spreadsheet budget.ods"


source = SpreadsheetSource(civiqhub.make_rows())
project = civiqhub.connect(source=source).project()
print(project.dataset.source, project.dataset.exposure)
spreadsheet budget.ods aggregated

Tests

File

Checks

tests/test_guarantees.py

The promises of this page: the core loads the standard library only, the token leaks nowhere, no personal data enters an object, no module computes a statistic

tests/test_api_source.py

The API source against a fake API, without network

tests/test_snippets.py

The snippets of the README and of Snippets, against the fake API

tests/test_docs_examples.py

Every other Python block of these pages, and the output it prints

tests/test_docs_reference.py

Every public module has a page in the reference

Words

Word

Meaning

dataset

Everything a source gave at one instant, frozen

actant

Whoever acts in a Project: a human member or an agent, known by a pseudonym only

vote counts

The five counts of the votes of a Revision, from score -2 to score +2

exposure

The level of detail served: aggregated or pseudonymised

placement

Where a Proposal sits in its parent article: a paragraph and a rank

citation

A Proposal shown in the text of an article that is not its parent article

impression

The fact that a Revision was shown to an Actant

deprecated value

A value of an old dataset that the model no longer has, read and marked, never produced

vote state

Where an Actant stands on a Revision: voted, abstained, not shown or unknown

fingerprint

16 hexadecimal characters of a SHA-256 of the rows; the same rows give the same fingerprint