Automating a script

[gisselletdean]

We're looking at running a web scraper for a site our company owns to pull rates for a table.
We need to run the web scraper every morning and need to see if there's a way to automate this process.

[DaneShayd]

In most cases, it would likely be best practice to host the web scraper on an external platform, scheduled to run at a certain time each day, then push the data into Hubspot via API.

[JasonByrne]

I’ve had a similar task with automating web scraping and found that setting it up in HubSpot was much easier than I thought. I used Puppeteer to automate form submissions, which really helped streamline the process. I needed to pull some data from a site daily, and instead of doing it manually, I used a script that would automatically submit a form and scrape the rates. It worked like a charm, and I could schedule the script to run every morning. I found this great blog post on submit form with puppeteer which guided me step-by-step through the process. It saved me so much time.