Nightly customer sync

A nightly sync keeps WorkWing’s customers in step with another system, and it can run either way. If your CRM or accounts package owns the customer list, you push: each night WorkWing is brought into line with it. If WorkWing is where customers are kept and you want them in a warehouse or report, you pull. This recipe covers pushing first, then pulling.

The examples use the workwing() helper from Handling errors, which retries safely and returns the response headers alongside data.

Pushing your customers into WorkWing

Match on your own customer number

Every customer has two fields for exactly this: external_crm_id holds your system’s customer number and external_crm_type names your system. Set both when you create a customer and you can find it again on every later run, whatever has happened to its name or email in the meantime.

WorkWing doesn’t enforce that external_crm_id is unique, and there’s no single call that creates-or-updates by it. The sync does the matching itself: it reads what WorkWing has, compares, and then creates or updates.

The key needs read:customers, create:customers and update:customers. It belongs to one brand, so a sync for two brands uses two keys.

1. Read what WorkWing already has

Read the whole set at the start of the run, 1,000 rows a page, rather than looking each customer up one at a time. Ten thousand customers is ten requests this way, against ten thousand the other way.

const FIELDS = [
  "name", "last_name", "email", "phone",
  "address_line1", "address_line2", "city", "postal_code", "country",
];

async function loadWorkWingCustomers() {
  const byCrmId = new Map();
  let lastId = null;

  for (;;) {
    const params = new URLSearchParams({
      external_crm_type: "eq.my-crm",
      select: ["id", "external_crm_id", ...FIELDS].join(","),
      order: "id.asc",
      limit: "1000",
    });
    if (lastId) params.set("id", `gt.${lastId}`);

    const { data, headers } = await workwing("GET", `/customers?${params}`);
    for (const c of data) byCrmId.set(c.external_crm_id, c);
    await pace(headers);

    if (data.length < 1000) return byCrmId;
    lastId = data[data.length - 1].id;
  }
}

This pages by id rather than by offset. Ordered by id and asking for the next page with id=gt. the last id you saw, a customer that changes or is deleted halfway through the read can’t shift the rows that follow it, so no row is skipped or read twice.

limit and offset work too, and suit a quick one-off walk:

GET /v1/customers?order=id.asc&limit=1000&offset=2000
Prefer: count=exact

With Prefer: count=exact the total comes back on the Content-Range header after the slash, so 2000-2999/10342 is the third page of 10,342. That’s handy for a progress bar. Read the header with headers.get("Content-Range").

2. Compare before you write

Only send a write when something has actually changed. A PATCH that changes nothing still spends a request from your allowance, and still moves the customer’s updated_at, so anything that reads changes by updated_at (your own pull, someone else’s report) sees every customer as changed every night.

function changedFields(source, existing) {
  const changes = {};
  for (const field of FIELDS) {
    const want = source[field] ?? null;
    const have = existing[field] ?? null;
    if (want !== have) changes[field] = want;
  }
  return changes;
}

Compare the values the way WorkWing stores them, or you’ll rewrite the same record every night. Phone numbers are the usual cause. WorkWing stores them in international form: if your brand’s country is the UK, 07700 900123 is stored as +447700900123, and anywhere else a number has to arrive with its country code. Convert your numbers to that form before comparing, and send them that way too.

3. Create, update or leave alone

async function pushCustomers(sourceCustomers) {
  const existing = await loadWorkWingCustomers();
  const counts = { created: 0, updated: 0, unchanged: 0, failed: 0 };

  for (const source of sourceCustomers) {
    const match = existing.get(source.crmId);
    const record = toWorkWing(source);

    try {
      if (!match) {
        const { headers } = await workwing("POST", "/customers", {
          body: { ...record, external_crm_id: source.crmId, external_crm_type: "my-crm" },
        });
        counts.created++;
        await pace(headers);
        continue;
      }

      const changes = changedFields(record, match);
      if (Object.keys(changes).length === 0) {
        counts.unchanged++;
        continue;
      }

      const { headers } = await workwing("PATCH", `/customers/${match.id}`, {
        body: changes,
      });
      counts.updated++;
      await pace(headers);
    } catch (err) {
      counts.failed++;
      console.error(source.crmId, err.code, err.message, err.requestId);
    }
  }
  return counts;
}

toWorkWing() is yours: it maps your record onto the WorkWing fields in FIELDS, already normalised. For a company, put the company name in name. For a person, put their first name in name and their surname in last_name. On a create, customer_type says which:

{ customer_type: "T_ORGANIZATION", name: "Acme Building Services" }
{ customer_type: "T_PERSON", name: "Priya", last_name: "Shah" }

The PATCH sends only the changed fields. Any field the endpoint doesn’t know is refused with E_AP_6002, so a typo shows up on the first run rather than being dropped silently.

One customer failing doesn’t stop the run. Log the code, the message and the request_id, carry on, and report the failures at the end. A bad email address or phone number is the usual reason, and it’ll fail the same way every night until someone fixes it at source.

Each write gets a fresh idempotency key from the helper, which covers retries inside the run. Re-running the whole sync is safe for a different reason: the run starts by reading what WorkWing has, so a customer created by an earlier run is found and compared, not created again.

Deleting

Decide deliberately whether your sync deletes anything. A customer who has any job in WorkWing can’t be deleted, so a sync that mirrors deletions will get refusals for exactly the customers your team has been working with. The safer default is to report customers that have gone from the source and leave the decision to a person.

Staying inside the rate limit

Your plan sets requests per minute per key: 60 on Free, 120 on Basic, 600 on Standard and 2,000 on Professional. On Free, a night with 3,000 changed customers is 3,000 writes, which takes 50 minutes at the limit. Comparing first is what keeps an ordinary night down to the records that really changed.

Slow down before the limit rather than bouncing off it:

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

async function pace(headers) {
  const remaining = Number(headers.get("X-RateLimit-Remaining"));
  const reset = Date.parse(headers.get("X-RateLimit-Reset"));
  if (remaining > 1 || Number.isNaN(reset)) return;
  await sleep(Math.max(0, reset - Date.now()) + 250);
}

X-RateLimit-Reset is a timestamp. A 429 still gets through now and then, for example when another job shares the key, and the helper waits out its Retry-After and tries again. Give the sync its own key so it doesn’t compete with your daytime integration for the same allowance.

Pulling WorkWing’s customers out

To copy customers into your own warehouse, read only the ones that changed since the last run. Every customer has an updated_at that moves whenever the customer record itself is edited, and you can filter on it.

async function pullChangedCustomers(since) {
  const until = new Date().toISOString();
  const rows = [];
  let lastId = null;

  for (;;) {
    const params = new URLSearchParams([
      ["updated_at", `gt.${since}`],
      ["updated_at", `lte.${until}`],
      ["order", "id.asc"],
      ["limit", "1000"],
    ]);
    if (lastId) params.append("id", `gt.${lastId}`);

    const { data, headers } = await workwing("GET", `/customers?${params}`);
    rows.push(...data);
    await pace(headers);

    if (data.length < 1000) break;
    lastId = data[data.length - 1].id;
  }

  return { rows, nextSince: until };
}

Store nextSince and pass it as since next night. Bounding the run with until means a customer edited while the pull is running is left for the next run rather than half-read in this one. Start the next window a minute before nextSince and write rows into your warehouse by id, so an overlap does no harm and an edit that landed right at the boundary isn’t missed.

What updated_at doesn’t tell you:

• Deleted customers. A deleted customer is simply gone, so it never appears as changed. Subscribe to the customer.deleted webhook, or compare the full list of ids now and then.
• Sites. A customer’s sites are separate records with their own updated_at. Pull /v1/locations the same way if you need addresses beyond the ones on the customer.

If you’d rather not wait for the night at all, the customer.created, customer.updated and customer.deleted webhooks tell you as each change happens. A nightly pull is still a good safety net behind them, for anything a webhook outage made you miss.

What’s next

• Handling errors: the helper these examples use, and which failures to retry
• Webhooks: hear about customer changes as they happen