Smarter Infinite Scroll With Automated Clicks

Scrape dynamic pages easily and reliably with Nimble’s new click_on_selector and idle_timeout parameters.

‍

Scraping infinite scroll pages has always been a challenge: you scroll, wait, and sometimes hit a “Load more” button just to keep data flowing. With Nimble’s latest Web API upgrade, that friction disappears. You can now automate clicks between scrolls and detect when content stops loading, making your pipelines faster, cleaner, and easier to manage.

Why We Built It

We heard it over and over from our users:

• Pages like Booking.com, Etsy, and search listings slow scraping jobs with manual clicks
• Endless scroll often leads to wasted cycles when no more content is available
• Customers were losing precious time dealing with infinite scrolling pages

With click_on_selector and idle_timeout, we’re giving engineers smart, reliable tools to navigate dynamic sites — with less code, fewer headaches, and faster deployment.

How to Use It

In this example, we use infinite scroll to define a duration, idle timeout, and the load more selector for booking.com:

curl -X POST 'https://api.webit.live/api/v1/realtime/web' \
--header 'Authorization: Basic <credential string>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "url": "https://www.booking.com/searchresults.en-gb.html?ss=Dizengoff+Square&checkin=2025-06-18&checkout=2025-06-19&group_adults=2&no_rooms=1&group_children=0",
    "format": "json",
    "parse": true,
    "render": true,
    "country": "US",
    "render_options": {
        "timeout": 120000
    },
    "render_flow": [
        {
            "wait": {
                "delay": 5000
            }
        },
        {
            "infinite_scroll": {
                "duration": 120000,
                "idle_timeout": 10000,
                "click_on_selector": "button[type=button].d0a01e3d83> span.ca2ca5203b"
            }
        }
    ]
}'

‍

See full documentation on infinite scrolling →

Real-World Example: Scraping Booking.com Listings

The challenge:

A user wants to scrape dozens of hotel listings but the page only loads ~20 results at a time, requiring “Load more” button clicks.

Without the feature:

You would have to write custom click handlers and figure out when to stop scrolling, adding fragile code to your pipeline.

With the feature:

You simply pass the button’s CSS selector into click_on_selector, set a reasonable idle_timeout, and let Nimble handle the rest. When the page stops changing height, the API knows it’s time to stop.

Result:

Cleaner jobs, less manual work, and faster extraction.

When to Use It

• Scraping sites with “Load more” buttons between scrolls
• Avoiding infinite loops on lazy-loaded content
• Optimizing scraping costs by stopping jobs as soon as data finishes loading

Best Practices

• Identify the right CSS selector for the “Load more” button; inspect the page carefully
• Tune idle_timeout based on how long content usually takes to load
• Combine with other render_flow actions like wait or scroll_to for full control

Scrape Smarter. Build Faster. Scale Without Limits.

With just one API call, you can eliminate manual scroll handling, skip brittle workarounds, and ship production-ready scrapers faster. Nimble’s Web API gives you the speed, reliability, and ease of use your data pipelines deserve.

Get Started →