Data Sync

The engine that polls provider APIs on a schedule and stores the records in StackOne’s indexes.

Data Sync consumes credits from your allowance. See plans and pricing.

Data Sync polls a provider on a schedule you set and stores the records it returns in a StackOne index. You don’t read that index directly. Deep Query searches and aggregates the stored records for you, without calling the provider.

You configure Data Sync once per action on a Connector Profile. Every account on that profile is then synced automatically.

How it works

• You turn sync on for an action on a Connector Profile, and set how often it runs.
• StackOne creates one sync per Linked Account on that profile, for accounts already linked and any linked later.
• Each time a sync fires it is a run, which fetches records from the provider and writes them to the index.

Turn on sync for an action

1. Open the Connector Profile

   In Connectors, open the connector and select the profile’s card. Data Sync is configured on a saved profile, not while you are creating one, and the action has to be switched on in that profile.

2. Find a syncable action

   On the Actions tab, actions that can be synced carry a Data Sync tag and a sync button next to their switch. Actions that need parameters to run, such as fetching one record by ID, can’t be synced.

3. Enable sync and set its cadence

   1. Click the sync button to open the Data Sync panel.
   2. Turn on Enable Data Sync.
   3. Set Sync every and Full re-sync.
   4. To turn on sync for every action on the profile that supports it, turn on Apply to all Actions.

   To be notified about synced changes from the same panel, see Data Sync Events.

4. Save

   Click Save Changes. The first sync starts immediately for every Linked Account already on the profile.

The Data Sync panel for one action

Field	What it controls	Bounds
Sync every	How often StackOne fetches new and changed records. Actions that can’t sync incrementally re-fetch everything on every run	15 minutes to 365 days
Full re-sync	How often StackOne re-fetches everything from scratch, to catch deletions and repair drift. Only shown for actions that support incremental syncs	At least 1 day, and less frequent than Sync every

Accounts linked after you save start syncing on their own first run, with no further configuration. You don’t need to revisit the profile as your customers link accounts.

Full and incremental runs

Most runs are incremental, because they’re cheap. Full runs happen on their own interval to catch what an incremental run structurally cannot.

	Incremental run	Full run
Fetches	Only records that changed since the last successful run, using a checkpoint the previous run left behind	Every record
Runs	Every Sync every interval	The first run, then every Full re-sync interval, and whenever the connector’s sync definition changes
Writes	Into the current set of records, in place	A complete new set, swapped in only once the run succeeds, so a query never sees a half-written result
Deletions at the provider	Not detected	Reflected
If it fails	The checkpoint is unchanged, so the next run covers the same window again rather than skipping records	The previous records stay readable

Deletions only surface on a full run. A record deleted at the provider stays readable until the next full re-sync, because an incremental run has no way to learn that it’s gone. Set Full re-sync to match how quickly you need deletions reflected.

Monitor syncs

Go to Accounts, open an account, and choose the Data Sync tab. Each synced action gets its own panel showing its cadence and when it next runs. Expand a sync to see its recent runs: whether each was full or incremental, how long it took, and how many records it created, updated, left unchanged, or deleted.

The Enabled toggle is the one setting that’s per-account. Turn it off to stop syncing an action for this account without touching the profile or any other account.

Run status	Meaning
success	The run finished and its records are readable
running	The run is in progress. The previously synced records stay readable until it finishes
failed	The run stopped on an error, shown in the row. The checkpoint is unchanged, so the next run retries the same window
truncated	The run stopped early at the page cap, or because the provider returned the same page twice
quota_exceeded	The run stopped early, or didn’t start, because the organization reached its record or storage limit
incomplete	The run stopped without reporting a result. The next scheduled run resumes from the last checkpoint

A sync shows Paused when the action, or sync for that action, is switched off on the Connector Profile. Nothing runs until it’s turned back on.

Deeply nested data

Data Sync stores every field a record has, however deeply it’s nested, but Deep Query only indexes fields up to four levels deep. Each object a field sits inside counts as one level and each list of objects as two, so address.city is one level deep and employment.history[].role is three.

Fields five or more levels deep, and lists of lists at any depth, are stored with the record without being indexed:

• They come back with the record unless source leaves them out, and a change to one of them counts as a change to the record.
• They can’t be used to filter, sort, or aggregate.
• A field five or more levels deep can still be named in source. The response then includes the whole nested object that holds it.

Each record stores up to 64 KB of this unindexed data. Past that, the largest nested parts are left out of the stored record first.

Data Sync events

A sync can also send an event to a webhook for every record a run creates, updates, or deletes, so you hear about changes without querying for them. See Data Sync Events.

Limits

Limit	Default
Records stored per organization	1,000,000
Storage per organization	25 GiB
Providers synced per organization	100
Pages fetched in one run	2,000
Duration of one run	6 hours
Record retention after the run that wrote it	30 days
Deeply nested data stored per record	64 KB

The provider count means providers with at least one sync enabled, not actions. Syncing a second action on a provider you already sync costs nothing against it, and two Connector Profiles on the same provider count once.

A connector can lower the per-run page and duration ceilings for an action that needs it, but never raise them.

The page cap is usually hit first. In most cases, a run reaches 2,000 pages before it reaches 6 hours, so the duration ceiling is a backstop rather than the limit a run normally stops on.

Turning on sync for an action fails if it would take the organization past 100 providers. The record and storage limits are checked as runs go, and a run that hits one stops early.

Two more constraints are worth planning around rather than reading as numbers:

• Not every action can be synced. Actions that need parameters to run, such as fetching one record by ID, have no sync. See Turn on sync for an action.
• Deletions wait for a full run. Set Full re-sync to match how quickly you need them reflected.

Records expire once that retention window passes without a run. Any enabled sync refreshes them well inside it, so expiry matters mainly for a sync you disable and leave off: its records drop out, and reads against them start reporting nothing synced.

Runs that stop early

A run stops early when it reaches the page cap, when the provider returns the same page twice, or when the organization reaches its record or storage limit. What happens next depends on the run:

• A full run is held. Its records are discarded, the last complete sync stays readable, and no deletions are recorded. The next full run is attempted on the Full re-sync cadence. The run shows truncated or quota_exceeded.
• An incremental run keeps what it wrote, but its checkpoint doesn’t move, so the next run covers the same window again.

Neither sends events, so a day with an early-stopped run can have no events for that action.

Next steps

Deep Query

Search, filter, and aggregate the records a sync has stored.

Connector Profiles

Where sync is configured, alongside actions and events.

Data Sync Events

Send an event to a webhook for every record a run creates, updates, or deletes.