Skip to main content

Dagster & Census (Component)

Community integration

This is a community-maintained integration. To report bugs or leave feedback, open an issue in the Dagster community integrations repo.

The dagster-census library provides a CensusComponent which can be used to easily represent Census syncs as assets in Dagster.

info

CensusComponent is a state-backed component, which fetches and caches Census workspace metadata. For information on managing component state, see Configuring state-backed components.

Step 1: Prepare a Dagster project​

To begin, you'll need a Dagster project. You can use an existing components-ready project or create a new one:

uvx create-dagster project my-project && cd my-project/src

Activate the project virtual environment:

source ../.venv/bin/activate

Finally, add the dagster-census library to the project:

uv add dagster-census

Step 2: Scaffold a Census component definition​

Now that you have a Dagster project, you can scaffold a Census component definition. You'll need to provide your Census API key, which you can set as an environment variable on the command line:

dg scaffold defs dagster_census.CensusComponent census_ingest

The dg scaffold defs call will generate a defs.yaml file:

.
└── census_ingest
    └── defs.yaml

YAML configuration​

In its scaffolded form, the defs.yaml file contains the configuration for your Census workspace:

my_project/defs/census_ingest/defs.yaml
type: dagster_census.CensusComponent

attributes:
  workspace:
    api_key: '{{ env.CENSUS_API_KEY }}'

Step 3: Check the component configuration​

You can check the configuration of your component with dg list defs:

$ dg list defs
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section   ┃ Definitions                                                                                 ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets    │ ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│           │ ┃ Key               ┃ Group   ┃ Deps ┃ Kinds   ┃ Description                              ┃ │
│           │ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│           │ │ census_sync_1     │ default │      │ census  │ Asset generated from Census sync 12345   │ │
│           │ ├───────────────────┼─────────┼──────┼─────────┼──────────────────────────────────────────┤ │
│           │ │ census_sync_2     │ default │      │ census  │ Asset generated from Census sync 67890   │ │
│           │ └───────────────────┴─────────┴──────┴─────────┴──────────────────────────────────────────┘ │
└───────────┴─────────────────────────────────────────────────────────────────────────────────────────────┘

Step 4: Select specific syncs​

You can select specific Census syncs to include in your component using the sync_selector key. This allows you to filter which syncs are represented as assets:

my_project/defs/census_ingest/defs.yaml
type: dagster_census.CensusComponent

attributes:
  workspace:
    api_key: "{{ env.CENSUS_API_KEY }}"
  sync_selector:
    by_name:
      - census_sync_1
$ dg list defs
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section   ┃ Definitions                                                                                 ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets    │ ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ │
│           │ ┃ Key               ┃ Group   ┃ Deps ┃ Kinds   ┃ Description                              ┃ │
│           │ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │
│           │ │ census_sync_1     │ default │      │ census  │ Asset generated from Census sync 12345   │ │
│           │ └───────────────────┴─────────┴──────┴─────────┴──────────────────────────────────────────┘ │
└───────────┴─────────────────────────────────────────────────────────────────────────────────────────────┘

You can also select syncs by ID:

my_project/defs/census_ingest/defs.yaml
type: dagster_census.CensusComponent

attributes:
  workspace:
    api_key: "{{ env.CENSUS_API_KEY }}"
  sync_selector:
    by_id:
      - 12345
      - 67890

About Census​

Census syncs data from your cloud warehouse to the SaaS tools your organization uses. It allows everyone in your organization to take action with good data, no custom scripts or API integrations required.