civiqhub.sources

Where data comes from: the port through which rows enter the SDK.

A Source has one duty: give rows in the format FORMAT, at the level of exposure asked for. A Dataset is then built from them. MemorySource serves rows held in memory, and FileSource a file written by save(). ApiSource reads the public API of CiviqHub with a personal access token; it is provisional, since the API is not published yet.

To write your own source:

  1. Derive a class from Source and give it a short name.

  2. Write its fetch(container=None, *, exposure). It returns rows in the format FORMAT, whose key exposure is the level asked for. at_exposure() serves rows held at one level at the level asked for.

  3. Write its describe(): one line that says where the rows come from, never a secret.

Then Dataset(source.fetch(exposure=Exposure.AGGREGATED), source=source.describe()) builds a dataset from it, and nothing in the package changes.

civiqhub.sources.DEFAULT_URL = 'https://api.civiqhub.com/v1'

The base address of the public API. Provisional: the API is not published yet.

civiqhub.sources.TOKEN_PREFIX = 'civ_pat_'

How every personal access token of CiviqHub starts.

civiqhub.sources.TOKEN_VARIABLE = 'CIVIQHUB_TOKEN'

The environment variable that holds the token, when none is passed.

class civiqhub.sources.ApiSource(token=None, *, base_url=DEFAULT_URL, transport=None, timeout=30.0)[source]

Bases: Source

The public API of CiviqHub, read with a personal access token. It only sends GET.

A token covers one container, a Project or a Group, and never gives more than what its owner sees. fetch() reads everything the token covers at one instant: it asks the API what the token covers and when (as_of), then passes that instant to every other read, so that all of them describe the same moment.

Provisional. The public API is not published. The base address, the paths (PATHS), the read that says what a token covers, the pagination and the reading at an instant are assumptions taken from the prototype. They live in this class only, so that they change without touching anything else.

The token is kept private. describe() and repr show its last four characters at most, and no message of an error holds it.

Parameters:
  • token (str | None) – The personal access token. Without it, the variable TOKEN_VARIABLE is read.

  • base_url (str) – The base address of the API.

  • transport (Callable[[str, Mapping[str, str]], tuple[int, Mapping[str, str], bytes]] | None) – What sends the requests. Without it, urllib.request sends them, with the delay timeout. Tests pass a fake one, so that no request leaves.

  • timeout (float) – How many seconds to wait for an answer.

Raises:
  • AuthenticationError – No token is given and the variable is not set, or the token does not start with TOKEN_PREFIX. The message never repeats it.

  • ValidationError – The base address is not an http or https address, or the delay is not a positive number.

name: ClassVar[str] = 'api'

A short word for the kind of source, such as "memory".

PATHS: ClassVar[dict[str, str]] = {'actants': '/projects/{id}/actants', 'amendments': '/projects/{id}/amendments', 'arguments': '/projects/{id}/arguments', 'group': '/groups/{id}', 'group_projects': '/groups/{id}/projects', 'participation': '/projects/{id}/participation', 'project': '/projects/{id}', 'proposals': '/projects/{id}/proposals', 'revisions': '/projects/{id}/revisions', 'scope': '/token', 'vote_counts': '/projects/{id}/votes/aggregates', 'votes': '/projects/{id}/votes'}

The path of each read, under the base address. {id} is the id of a Group or of a Project. A read of pages answers {"data": [...], "next_cursor": ...} and is read again with the parameter cursor until no cursor comes back.

  • scope: what the token covers, {"kind": "project" or "group", "id": ..., "as_of": ...};

  • group, project, participation: one row;

  • group_projects, proposals, revisions, amendments, arguments, vote_counts: pages;

  • actants, votes: pages, at the pseudonymised level only.

base_url: str

The base address of the API, without a final slash.

timeout: float

How many seconds to wait for an answer.

describe()[source]

Return "api <base address>, token civ_pat_...<last four characters>".

A token so short that four characters would show half of its secret part shows fewer.

Return type:

str

fetch(container=None, *, exposure)[source]

Read everything the token covers, at one instant, at the level asked for.

Parameters:
  • container (str | None) – The id or the slug of one Project of the Group that the token covers. Without it, every Project the token covers is read.

  • exposure (Exposure) – The level asked for. At the pseudonymised level, the votes and the actants are read too, and the actants of several Projects are merged by pseudonym.

Returns:

Rows in the format FORMAT. A link to a Group that the token does not cover, such as the Group of a Project read with a token of that Project, is left out, so that the rows hold no link to a missing row.

Raises:
  • AuthenticationError – The API refused the token: it is expired or revoked.

  • ExposureError – The token does not allow a read, such as the votes.

  • NotFoundError – The API has nothing at a path, or container names no Project that the token covers.

  • RateLimitedError – The API asks to slow down; retry_after says how long.

  • CiviqHubError – The API cannot be reached, or answered something else.

Return type:

dict[str, Any]

_to_rows(section, answer)[source]

Turn an answer of the API into rows of a section of the format.

It is the only place where the shape of the answers meets the format of the rows. Today, the API is assumed to answer rows already.

Parameters:
  • section (str)

  • answer (Any)

Return type:

list[dict[str, Any]]

class civiqhub.sources.FileSource(path)[source]

Bases: Source

A file written by save(), read without network.

The file is read at each fetch(). It covers what its rows cover, so container is ignored.

Parameters:

path (Path) – The path of the file.

name: ClassVar[str] = 'file'

A short word for the kind of source, such as "memory".

path: Path

The path of the file.

describe()[source]

Return "file <name of the file>".

Return type:

str

fetch(container=None, *, exposure)[source]

Read the file and return its rows, at the level asked for.

Parameters:
  • container (str | None) – Ignored: the source covers what its file covers.

  • exposure (Exposure) – The level asked for.

Raises:
Return type:

dict[str, Any]

class civiqhub.sources.MemorySource(rows)[source]

Bases: Source

Rows held in memory: for tests, simulations and the data of another system.

The source keeps a deep copy of the rows: changing them afterwards does not change what it gives. It covers what the rows cover, so container is ignored.

Parameters:

rows (Mapping[str, Any]) – The rows, in the format FORMAT.

Raises:

ValidationError – The rows are not a dictionary.

name: ClassVar[str] = 'memory'

A short word for the kind of source, such as "memory".

fetch(container=None, *, exposure)[source]

Return a copy of the rows, at the level asked for (see at_exposure()).

Parameters:
  • container (str | None) – Ignored: the source covers what its rows cover.

  • exposure (Exposure) – The level asked for.

Raises:

ExposureError – The rows are aggregated and the pseudonymised level is asked for.

Return type:

dict[str, Any]

class civiqhub.sources.Source[source]

Bases: ABC

Anything that gives rows in the format of a dataset: the API, a file, memory.

A source is the first point of extension of the SDK: a class derived from it works with Dataset as the sources of the package do. The docstring of civiqhub.sources says how to write one.

name: ClassVar[str] = 'source'

A short word for the kind of source, such as "memory".

abstractmethod fetch(container=None, *, exposure)[source]

Return the rows of what the source covers, at the level of exposure asked for.

Parameters:
  • container (str | None) – The id or the slug of one Project, when the source covers several. Without it, the rows cover everything the source covers.

  • exposure (Exposure) – The level of detail of the rows. Their key exposure equals it.

Returns:

Rows in the format FORMAT.

Return type:

dict[str, Any]

describe()[source]

Return one line that says where the rows come from, never with a secret.

A Dataset shows it as its source. By default, it is name.

Return type:

str

civiqhub.sources.Transport

What sends a request for ApiSource: it receives an address and headers, sends a GET, and returns the status code, the headers and the body of the answer.

alias of Callable[[str, Mapping[str, str]], tuple[int, Mapping[str, str], bytes]]

civiqhub.sources.at_exposure(rows, exposure)[source]

Serve rows held at one level of exposure at the level asked for.

Rows never gain detail. Rows held at the level asked for come back as a copy. Pseudonymised rows asked for at the aggregated level come back reduced: without the sections actants, votes and impressions, without the key author_id in proposals and arguments, and with a row of vote_counts, counted from the votes, for each revision that has votes and no such row.

Parameters:
  • rows (Mapping[str, Any]) – The rows, in the format FORMAT. They are not changed.

  • exposure (Exposure) – The level asked for.

Returns:

New rows, at the level asked for.

Raises:
  • ExposureError – The rows are aggregated and the pseudonymised level is asked for.

  • ValidationError – The rows or the level are not valid, or a vote cannot be counted.

Return type:

dict[str, Any]