Human-in-the-loop browser automation
Human-in-the-loop browser automation lets software do the routine work while a person handles sign-ins, decisions and exceptions. In Pine Computer, a live desktop can appear inside your product. A person can watch, take the controls and hand them back, with one driver at a time.
By Pine AI · Published
Choose the handoff points before the run
Separate three situations: the computer needs private input, the job needs a decision, or the work has reached an action your application requires a person to approve. These situations need different instructions. A login asks someone to enter credentials; a purchase decision needs the item, total and delivery details.
For a shopping assistant, define a workflow that compares baskets and presents the total before purchase. Approval should identify the specific purchase. This is an application design pattern, not a claim that a model will enforce every business rule on its own.
Pine’s train-search recording gives a narrower example: find the cheapest day for two adults to travel from London to Paris, then return the day, time and price. The run stops before passenger or payment details. Set an equally concrete stopping point for your workflow, and distinguish the research result from any later authorization to book.
Separate watching from driving
An embedded live view lets a person see what is happening. Giving that person control is a separate action. Pine Computer has one driver at a time: the AI, your code or a person. Taking over stops the AI until the person hands back.
Your backend authorizes the handoff. Keep server credentials on the backend and use the scoped browser access described in the Web SDK and token documentation. Do not treat a visible desktop as authorization to drive it or let two drivers issue competing actions.
Make the handoff understandable
The person needs to know why the job paused and what will happen when they return control. Put that explanation beside the live screen. Keep the original request visible so they can judge whether the current page and account match it.
In Pine’s DocuSign example, the agent opens an NDA and fills the fields, with an explicit instruction to leave the signature to the user. After the run stops, the application notifies the person and waits while they sign in the live view. The application supplies the notification channel; embedding a desktop alone does not send a text or make a call.
- Pause the work and explain the exact input or decision needed.
- Show the relevant page, record and proposed action.
- Let the authorized person take control or decline the request.
- Require an explicit handback before the AI continues.
- Check the resulting state and return an outcome to the application.
Handle private input deliberately
A password or verification code is different from an instruction for the model. Pine Computer documents blind sessions for input a person should enter without the AI seeing it. Use the documented mechanism and check the current security behavior before integrating it into a sensitive workflow.
Sign-ins can persist per computer, but the application must still handle expired authentication. The computer’s saved state is not an archive of every working file. Retrieve the artifacts your product needs and keep the customer’s account boundary consistent across later jobs.
Test the exceptions as part of the product
Try an approval that is declined, a person who leaves before handing back, an expired login and a page that changes while the job is paused. Define what your product shows in each case and whether it can resume safely or needs a fresh request.
Measure how often people intervene and whether their intervention resolves the job. A workflow that pauses clearly and returns an honest unresolved result can be more useful than one that reports success without sufficient evidence. The examples below are illustrative workflows; Pine Computer is in private beta.
Explore the workflow
These use cases are examples, not customer case studies.
- Compare and buy: Your assistant finds the cheapest place to buy it, delivery included, and places the order when the person says yes.
- Book an appointment: Asked for a check-up before ten, your assistant books the first open slot the person agrees to.
- Cancel a subscription: Every “before you go” offer is noted, and the plan is still cancelled before it renews.
Product documentation
Use the developer docs for current API contracts and integration instructions.