civiqhub.sources¶
Where data comes from: the port through which rows enter the SDK.
A Source has one duty: give rows in the format FORMAT,
at the level of exposure asked for. A Dataset is then built
from them. MemorySource serves rows held in memory, and FileSource a
file written by save(). ApiSource reads the public
API of CiviqHub with a personal access token; it is provisional, since the API is not
published yet.
To write your own source:
Derive a class from
Sourceand give it a shortname.Write its
fetch(container=None, *, exposure). It returns rows in the formatFORMAT, whose keyexposureis the level asked for.at_exposure()serves rows held at one level at the level asked for.Write its
describe(): one line that says where the rows come from, never a secret.
Then Dataset(source.fetch(exposure=Exposure.AGGREGATED), source=source.describe())
builds a dataset from it, and nothing in the package changes.
- civiqhub.sources.DEFAULT_URL = 'https://api.civiqhub.com/v1'¶
The base address of the public API. Provisional: the API is not published yet.
- civiqhub.sources.TOKEN_PREFIX = 'civ_pat_'¶
How every personal access token of CiviqHub starts.
- civiqhub.sources.TOKEN_VARIABLE = 'CIVIQHUB_TOKEN'¶
The environment variable that holds the token, when none is passed.
- class civiqhub.sources.ApiSource(token=None, *, base_url=DEFAULT_URL, transport=None, timeout=30.0)[source]¶
Bases:
SourceThe public API of CiviqHub, read with a personal access token. It only sends GET.
A token covers one container, a Project or a Group, and never gives more than what its owner sees.
fetch()reads everything the token covers at one instant: it asks the API what the token covers and when (as_of), then passes that instant to every other read, so that all of them describe the same moment.Provisional. The public API is not published. The base address, the paths (
PATHS), the read that says what a token covers, the pagination and the reading at an instant are assumptions taken from the prototype. They live in this class only, so that they change without touching anything else.The token is kept private.
describe()andreprshow its last four characters at most, and no message of an error holds it.- Parameters:
token (str | None) – The personal access token. Without it, the variable
TOKEN_VARIABLEis read.base_url (str) – The base address of the API.
transport (Callable[[str, Mapping[str, str]], tuple[int, Mapping[str, str], bytes]] | None) – What sends the requests. Without it,
urllib.requestsends them, with the delaytimeout. Tests pass a fake one, so that no request leaves.timeout (float) – How many seconds to wait for an answer.
- Raises:
AuthenticationError – No token is given and the variable is not set, or the token does not start with
TOKEN_PREFIX. The message never repeats it.ValidationError – The base address is not an
httporhttpsaddress, or the delay is not a positive number.
- name: ClassVar[str] = 'api'¶
A short word for the kind of source, such as
"memory".
- PATHS: ClassVar[dict[str, str]] = {'actants': '/projects/{id}/actants', 'amendments': '/projects/{id}/amendments', 'arguments': '/projects/{id}/arguments', 'group': '/groups/{id}', 'group_projects': '/groups/{id}/projects', 'participation': '/projects/{id}/participation', 'project': '/projects/{id}', 'proposals': '/projects/{id}/proposals', 'revisions': '/projects/{id}/revisions', 'scope': '/token', 'vote_counts': '/projects/{id}/votes/aggregates', 'votes': '/projects/{id}/votes'}¶
The path of each read, under the base address.
{id}is the id of a Group or of a Project. A read of pages answers{"data": [...], "next_cursor": ...}and is read again with the parametercursoruntil no cursor comes back.scope: what the token covers,{"kind": "project" or "group", "id": ..., "as_of": ...};group,project,participation: one row;group_projects,proposals,revisions,amendments,arguments,vote_counts: pages;actants,votes: pages, at the pseudonymised level only.
- base_url: str¶
The base address of the API, without a final slash.
- timeout: float¶
How many seconds to wait for an answer.
- describe()[source]¶
Return
"api <base address>, token civ_pat_...<last four characters>".A token so short that four characters would show half of its secret part shows fewer.
- Return type:
str
- fetch(container=None, *, exposure)[source]¶
Read everything the token covers, at one instant, at the level asked for.
- Parameters:
container (str | None) – The id or the slug of one Project of the Group that the token covers. Without it, every Project the token covers is read.
exposure (Exposure) – The level asked for. At the pseudonymised level, the votes and the actants are read too, and the actants of several Projects are merged by pseudonym.
- Returns:
Rows in the format
FORMAT. A link to a Group that the token does not cover, such as the Group of a Project read with a token of that Project, is left out, so that the rows hold no link to a missing row.- Raises:
AuthenticationError – The API refused the token: it is expired or revoked.
ExposureError – The token does not allow a read, such as the votes.
NotFoundError – The API has nothing at a path, or
containernames no Project that the token covers.RateLimitedError – The API asks to slow down;
retry_aftersays how long.CiviqHubError – The API cannot be reached, or answered something else.
- Return type:
dict[str, Any]
- _to_rows(section, answer)[source]¶
Turn an answer of the API into rows of a section of the format.
It is the only place where the shape of the answers meets the format of the rows. Today, the API is assumed to answer rows already.
- Parameters:
section (str)
answer (Any)
- Return type:
list[dict[str, Any]]
- class civiqhub.sources.FileSource(path)[source]¶
Bases:
SourceA file written by
save(), read without network.The file is read at each
fetch(). It covers what its rows cover, socontaineris ignored.- Parameters:
path (Path) – The path of the file.
- name: ClassVar[str] = 'file'¶
A short word for the kind of source, such as
"memory".
- path: Path¶
The path of the file.
- fetch(container=None, *, exposure)[source]¶
Read the file and return its rows, at the level asked for.
- Parameters:
container (str | None) – Ignored: the source covers what its file covers.
exposure (Exposure) – The level asked for.
- Raises:
NotFoundError – No file is at the path.
ValidationError – The file is not JSON, or not a dictionary of sections.
ExposureError – The file is aggregated and the pseudonymised level is asked for.
- Return type:
dict[str, Any]
- class civiqhub.sources.MemorySource(rows)[source]¶
Bases:
SourceRows held in memory: for tests, simulations and the data of another system.
The source keeps a deep copy of the rows: changing them afterwards does not change what it gives. It covers what the rows cover, so
containeris ignored.- Parameters:
rows (Mapping[str, Any]) – The rows, in the format
FORMAT.- Raises:
ValidationError – The rows are not a dictionary.
- name: ClassVar[str] = 'memory'¶
A short word for the kind of source, such as
"memory".
- fetch(container=None, *, exposure)[source]¶
Return a copy of the rows, at the level asked for (see
at_exposure()).- Parameters:
container (str | None) – Ignored: the source covers what its rows cover.
exposure (Exposure) – The level asked for.
- Raises:
ExposureError – The rows are aggregated and the pseudonymised level is asked for.
- Return type:
dict[str, Any]
- class civiqhub.sources.Source[source]¶
Bases:
ABCAnything that gives rows in the format of a dataset: the API, a file, memory.
A source is the first point of extension of the SDK: a class derived from it works with
Datasetas the sources of the package do. The docstring ofciviqhub.sourcessays how to write one.- name: ClassVar[str] = 'source'¶
A short word for the kind of source, such as
"memory".
- abstractmethod fetch(container=None, *, exposure)[source]¶
Return the rows of what the source covers, at the level of exposure asked for.
- Parameters:
container (str | None) – The id or the slug of one Project, when the source covers several. Without it, the rows cover everything the source covers.
exposure (Exposure) – The level of detail of the rows. Their key
exposureequals it.
- Returns:
Rows in the format
FORMAT.- Return type:
dict[str, Any]
- civiqhub.sources.Transport¶
What sends a request for
ApiSource: it receives an address and headers, sends a GET, and returns the status code, the headers and the body of the answer.alias of
Callable[[str,Mapping[str,str]],tuple[int,Mapping[str,str],bytes]]
- civiqhub.sources.at_exposure(rows, exposure)[source]¶
Serve rows held at one level of exposure at the level asked for.
Rows never gain detail. Rows held at the level asked for come back as a copy. Pseudonymised rows asked for at the aggregated level come back reduced: without the sections
actants,votesandimpressions, without the keyauthor_idinproposalsandarguments, and with a row ofvote_counts, counted from the votes, for each revision that has votes and no such row.- Parameters:
- Returns:
New rows, at the level asked for.
- Raises:
ExposureError – The rows are aggregated and the pseudonymised level is asked for.
ValidationError – The rows or the level are not valid, or a vote cannot be counted.
- Return type:
dict[str, Any]