civiqhub.dataset¶
The data of a source at one instant, as one frozen graph of objects.
A Dataset is built from rows in the format FORMAT: a dictionary that JSON
can write, with an entry format, an instant as_of, a level of exposure and one
list of rows per section. A missing section is an empty list, and a key that a row does
not need is ignored.
This version builds every section of the format: groups, projects,
participation, articles, article_versions, paragraphs, placements,
citations, proposals, revisions, vote_counts, amendments,
arguments, and, in a pseudonymised dataset only, actants, votes and
impressions. A key that the format does not know is kept: it counts in the
fingerprint and comes back from Dataset.to_dict().
A row of citations has the keys of a row of placements: proposal_id,
article_id, version_number, paragraph_id and rank. It is a Proposal cited
in an article that is not its parent article.
An old dataset can hold values that the Articles model deprecated: the status DRAFT,
the types CREATE and REMOVE, the pending changes PENDING_CREATE and
PENDING_REMOVE, the kind SYNC and the traces of forks. They are read without
failing, and each one is marked by a DeprecatedValue, listed by
Dataset.deprecated. A row of amendments that is a creation, a removal or a
pending change is not a local change of a Proposal, so it builds no Amendment.
Dataset.save() writes the rows to a JSON file, and Dataset.from_file() reopens
it, with the same fingerprint.
- civiqhub.dataset.FORMAT = 'civiqhub.dataset/0'¶
The format of the rows that a
Datasetis built from and gives back.
- class civiqhub.dataset.Dataset(rows, *, source='memory')[source]¶
Bases:
objectEverything a source gave at one instant, at one level of exposure, frozen.
A dataset checks its rows once, when it is built, and then holds frozen objects: nothing can be added, removed or changed, and no navigation reads the network. Two datasets built from the same rows have the same
fingerprint(), but their objects differ, since an object of the product belongs to its dataset.- Parameters:
rows (Mapping[str, Any]) – The rows, in the format
FORMAT. They are copied: changing them afterwards does not change the dataset.source (str) – Where the rows come from, such as
"memory". It is shown byreprand must never hold a secret.
- Raises:
ValidationError – The rows are not in the format
FORMAT, or break a rule of the model. The message names the row and the rule.
- classmethod from_dict(rows, *, source='memory')[source]¶
Build a dataset from rows in the format
FORMAT.It is the same as calling
Dataset.- Parameters:
rows (Mapping[str, Any]) – The rows. They are copied.
source (str) – Where the rows come from, never a secret.
- Raises:
ValidationError – The rows are not in the format, or break a rule of the model.
- Return type:
- classmethod from_file(path)[source]¶
Open a dataset that
save()wrote, without network and without token.The dataset that comes back has the same fingerprint as the one saved, and the source
"file <name of the file>".- Parameters:
path (str | Path) – The path of the file.
- Raises:
NotFoundError – No file is at this path.
ValidationError – The file is not JSON, or not in the format
FORMAT.
- Return type:
- property as_of: str¶
The instant that the data describes, in ISO 8601 and in UTC.
- property source: str¶
Where the rows come from, such as
"memory". It never holds a secret.
- property article_versions: Listing[ArticleVersion]¶
Every version of every article: by article, in the order of the rows, then by number.
- property paragraphs: Listing[Paragraph]¶
Every paragraph of every article version: by version, in the order of the text.
- property placements: Listing[Placement]¶
Every placement of a Proposal: by version, then in the order of the text and of the ranks.
- property deprecated: tuple[DeprecatedValue, ...]¶
Every deprecated value read in the rows, in the order of the sections and rows.
It is empty for a dataset of the current model. See
DeprecatedValuefor the values that are marked.
- property citations: Listing[Citation]¶
Every citation, in every article version: by version, then in the order of the text and of the ranks.
- property revisions: Listing[Revision]¶
Every Revision of the dataset, by Proposal, then by number.
The Proposals come in the order of their rows.
- property actants: Listing[Actant]¶
Every Actant of the dataset, in the order of the rows; empty when aggregated.
- property votes: Listing[Vote]¶
Every vote of the dataset, in the order of the rows; empty when aggregated.
- property has_impressions: bool¶
Whether the rows say which Revisions were shown to which Actants.
It is true when the rows hold the section
impressions, even empty.
- project(ref=None)[source]¶
Return the Project with this id or slug, or the only Project.
- Parameters:
ref (str | None) – The id or the slug of the Project. Without it, the dataset must hold exactly one Project.
- Raises:
NotFoundError – No Project has this id or slug, or
refis left out and the dataset holds no Project or several. The message lists up to eight slugs.- Return type:
- group(ref=None)[source]¶
Return the Group with this id or slug, or the only Group.
- Parameters:
ref (str | None) – The id or the slug of the Group. Without it, the dataset must hold exactly one Group.
- Raises:
NotFoundError – No Group has this id or slug, or
refis left out and the dataset holds no Group or several. The message lists up to eight slugs.- Return type:
- property has_individual_votes: bool¶
Whether the dataset holds individual votes, which the pseudonymised level does.
- require_individual_votes(what)[source]¶
Check that the dataset holds individual votes, before reading them.
- Parameters:
what (str) – What needs the individual votes, such as
"the vote matrix". The message of the error starts with it.- Raises:
ExposureError – The dataset is aggregated. Its hint says how to get the pseudonymised level.
- Return type:
None
- save(path)[source]¶
Write the rows of the dataset to a JSON file, which
from_file()reopens.The file holds exactly the rows, in UTF-8, with sorted keys: every section, including the articles and their versions as the source gave them. It holds neither a token nor the source. A pseudonymised file holds individual votes under pseudonyms: share it as the exposure policy of the Project allows, and never commit it to a public repository.
- Parameters:
path (str | Path) – Where to write the file. An existing file is replaced.
- Returns:
The path of the file.
- Return type:
Path
- fingerprint()[source]¶
Return a short digest of the rows: the same rows give the same fingerprint.
It is the first 16 hexadecimal characters of the SHA-256 of the rows written in JSON, with sorted keys and without spaces. Every section counts, including those that this version does not build. The source does not count.
- Return type:
str