Scheduled API Data Imports with a Monitoring Screen

I build scheduled data imports that show what arrived, what passed validation and what still needs a decision. An API is an agreed way for systems to exchange data. Before scheduling anything, I confirm that the source provides the required access and identify the records the receiving application should own.

This service focuses on recurring data retrieval and review. It is not a promise to connect every system or collect unrestricted data from arbitrary websites. The source, permitted access method and destination are part of the scope.

Example: A morning import of 126 records

In the fictional illustration, the job is scheduled for 08:00. It receives 126 records: 124 pass validation and two need review. The run finishes at 08:03 with that result. The completion time does not imply that the two exceptions were successfully imported; the review count remains visible.

Whether the 124 valid records can be published depends on the workflow. An independent record feed may allow partial publication. A report that requires a complete set may need the entire batch held. I make that rule explicit rather than letting an incidental error determine it. A corrected exception stays attached to its original import when it is retried.

Retrying must not multiply records

If the source returns data in pages, the import tracks its progress. Restarting an interrupted job should not create duplicate orders already received. Stable source identifiers, change timestamps and an import batch record support that behaviour. We also decide what a deletion at the source should mean at the destination.

Expired access, changed fields and usage limits are different failures. The screen should show an understandable cause and the last successful data time. If the application continues displaying older data during an outage, it needs to identify that data as older rather than calling it current.

When an existing automation works

Apps Script offers scheduled triggers, and Power Automate supports scheduled cloud flows. Either may suit a simple source and output. A dedicated screen adds value when operators need exception handling, multiple sources and controlled restarts in one place.

Making the scope testable

I start with a de-identified example of your current file or screen and the roles of the people using it. The written scope defines where each field comes from, who can change it and where the result goes. I test an interrupted page sequence, a repeated run and expired access. Accepted and review totals must account for the complete number of records received from the source.

The scope also covers exports, the backup approach and who handles failures. We review the first version against one real workflow before planning further connections.

Frequently Asked Questions

Will all the data be ready at 08:00?

A scheduled start and a completed import are different. Source response times and record volumes affect completion. We agree how delays are displayed and when someone should be notified.

Do two invalid records stop the other 124?

That depends on the agreed completeness rule. I distinguish partial publication from holding the entire batch, and the interface shows which approach was used.

What if the source has no API?

I assess authorised file exports or another supported connection. I do not promise recurring access before the source and its permitted retrieval method have been checked.

Discuss your workflow

For examples described in everyday business terms, see importing website data into Excel, building an Excel sales report or tracking product price changes. This page covers the technical scope of connecting to a data source on a schedule and checking the import.

For a related workflow, see Import Excel Attachments from Email. Explore custom web applications for your team or contact me with an example of the process you want to improve.