Why AI Web Agents Is the Next Infrastructure Layer for Insurance

There is a useful exercise for understanding where insurance technology is headed: look at where fintech was ten years ago.

In 2013, if you wanted to build a fintech app that accessed a user's bank account, you had two options. You could negotiate an API partnership with each bank individually — a process that took months or years per institution and was available from very few banks. Or you could use screen-scraping technology that logged into the bank's website programmatically, navigated the interface, and extracted the data.

Plaid, the company that became the backbone of fintech connectivity, started as a screen-scraper. It connected to bank websites the way a user would — entering credentials, navigating pages, reading balances and transactions from the rendered HTML. It was, in the technical sense, AI web agents. Over time, as banks began offering APIs, Plaid shifted to API connections where available and maintained browser-based connections where APIs didn't exist. Today, Plaid is valued at billions of dollars and is considered critical financial infrastructure.

Insurance is following the same pattern, roughly a decade behind.

TLDR: Fewer than 50 of the roughly 2,700 U.S. P&C carriers offer commercial quoting APIs. AI web agents — technology that interacts with carrier web portals programmatically — bridges this gap. The parallel with fintech is direct: Plaid built financial infrastructure on screen-scraping before banks offered APIs. Insurance AI web agents is the Plaid moment for carrier connectivity.

The API Coverage Problem

The insurance industry's connectivity challenge is well-documented but worth restating in precise terms, because the scale of the gap drives the infrastructure argument.

According to NAIC data, there are approximately 2,700 property and casualty carriers operating in the United States. Of those, fewer than 50 have built APIs that allow third-party tools to submit commercial insurance applications and retrieve quotes programmatically.

The carriers with APIs are predominantly the largest nationals: The Hartford, Progressive, Travelers, CNA, Liberty Mutual, Nationwide, and a handful of larger regionals like Acuity and Erie. Platforms like Tarmika and Semsee, which rely on API connections, each connect to roughly 31 to 48 carriers.

That leaves approximately 2,650 carriers — including most regional carriers, mutual companies, state-specific writers, and specialty markets — accessible only through their individual web portals.

Why the Gap Persists

The natural question is: why haven't more carriers built APIs? The answer involves economics, technology debt, and incentive structures.

The economics don't work for most carriers. Building and maintaining a commercial quoting API requires significant investment: integration with legacy policy administration systems, ongoing maintenance as rating algorithms change, compliance with state-specific filing requirements, and security infrastructure. For a regional carrier writing a few hundred million in premium, the return on this investment is marginal. The carrier already has agents logging into its portal and writing business.

Legacy systems resist API exposure. A majority of U.S. P&C carriers run their core operations on policy administration systems built in the 1990s or 2000s — platforms like Guidewire, Majesco, Duck Creek, or custom-built mainframe applications. Exposing rating logic through modern REST or GraphQL APIs means building an abstraction layer on top of these systems, which is technically complex and carries operational risk.

The pace of change is constant. Commercial insurance rates change with every state filing. Coverage forms are updated by ISO and AAIS. Underwriting guidelines shift quarterly. An API that reflects all of these changes in real time requires continuous maintenance — a staffing and engineering commitment that most carriers' IT departments are not resourced to sustain.

Carrier IT priorities lie elsewhere. When carrier CIOs allocate development resources, commercial quoting APIs compete with core system modernization, regulatory compliance, claims system upgrades, and internal tooling. API development for third-party agent tools rarely wins that prioritization contest.

The result is a connectivity gap that has existed for over a decade and shows no signs of closing through API adoption alone. Even optimistic projections suggest that over 95% of carriers will remain portal-only for the foreseeable future.

The AI Web Agents Taxonomy

Not all AI web agents is the same. The technology landscape includes several distinct approaches, each with different tradeoffs:

Traditional RPA (Robotic Process Automation)

Traditional RPA platforms — UiPath, Automation Anywhere, Blue Prism — were the first generation of commercial AI web agents. These tools record and replay human interactions with desktop applications and web interfaces. An operator performs a task manually while the RPA tool records each click, keystroke, and field entry. The recording is then parameterized and can be replayed with different input data.

Strengths: Mature technology, large vendor ecosystem, works with both web and desktop applications, well-understood by enterprise IT teams.

Limitations: Brittle. Recordings break when the target application's UI changes — a moved button, a renamed field, a restructured page. Insurance carrier portals change frequently (quarterly or more), which means RPA recordings require constant maintenance. RPA is also typically slow, running at near-human speed because it simulates mouse movements and keystrokes rather than interacting with the page's underlying structure.

Infrastructure fitness: Low. Traditional RPA works as a point automation for specific, stable workflows. It doesn't scale well as infrastructure because the maintenance burden grows linearly with the number of carriers and the frequency of portal changes.

Headless AI Web Agents

Modern headless browser frameworks — Playwright (Microsoft), Puppeteer (Google), Selenium — interact with web pages at the DOM (Document Object Model) level rather than simulating mouse movements. They can navigate pages, fill forms, click buttons, and extract data by referencing HTML elements directly, without rendering the visual interface.

Strengths: Faster than RPA (no visual rendering overhead), more precise (interacting with DOM elements rather than screen coordinates), better error handling (can inspect page state programmatically), natively parallel (can run multiple browser instances simultaneously).

Limitations: Requires engineering talent to build and maintain. Each carrier portal integration is essentially a custom software project. Portal changes still require updates, though the failure mode is more graceful (an element not found) rather than catastrophic (clicking the wrong thing).

Infrastructure fitness: Medium-high. Headless AI web agents can scale to hundreds of carrier integrations if paired with robust portal monitoring and maintenance processes. The parallel execution capability is critical for multi-carrier quoting workflows where time matters.

AI-Driven Browser Agents

The newest category uses large language models (LLMs) and computer vision to navigate web interfaces without pre-programmed instructions for each portal. Instead of recording a workflow or writing DOM selectors, an AI agent "looks" at the portal (via screenshots or DOM analysis), understands the form's intent, and fills it out based on the input data and its understanding of insurance concepts.

Strengths: Potentially self-healing — can adapt to portal changes without explicit reprogramming. Can navigate portals it has never encountered before if they follow standard insurance quoting patterns. Reduces per-carrier integration cost.

Limitations: Current reliability is below production-grade for insurance transactions where accuracy is critical. Hallucination risk — an AI agent that fills a wrong field or enters incorrect data creates E&O liability. Speed is variable and generally slower than optimized headless automation. Cost per transaction (LLM inference costs) is higher than deterministic automation.

Infrastructure fitness: High potential, currently medium. AI-driven browser agents are likely the long-term evolution of carrier connectivity, but they need to cross a reliability threshold before they can serve as trusted infrastructure. The insurance industry's tolerance for error in quoting and binding transactions is extremely low.

Hybrid Approaches

In practice, the most effective production systems use hybrid approaches:

This layered architecture mirrors how infrastructure typically evolves: deterministic systems handle the known cases efficiently, while adaptive systems handle exceptions and changes.

Production Challenges: Why This Is Harder Than It Looks

Building AI web agents that works in a demo is straightforward. Building AI web agents that works reliably at production scale across dozens of carrier portals is an engineering challenge of a different order. The gap between "it works" and "it works every time" is where most automation projects fail.

CAPTCHA and Bot Detection

Carrier portals increasingly deploy CAPTCHA challenges and bot detection mechanisms. These range from simple image CAPTCHAs to sophisticated behavioral analysis that detects automated interaction patterns (consistent timing between keystrokes, lack of mouse movement variance, unusual navigation patterns).

Production-grade automation must handle these challenges without slowing down the quoting workflow. Approaches include CAPTCHA-solving services, behavioral mimicry (adding human-like variance to interaction patterns), and in some cases, pre-authentication arrangements with carriers.

Session Management

Insurance quoting workflows involve multi-page sessions that can take several minutes per carrier. Each session must maintain state across pages — data entered on page one must persist through page five. Sessions can time out, pages can fail to load, and server-side validation can reject inputs in ways that require backtracking.

At scale, a platform quoting 15 carriers simultaneously is managing 15 concurrent browser sessions, each with its own state, its own error conditions, and its own timing characteristics. This is a distributed systems problem, not a simple automation problem.

Credential Management

AI web agents uses the agent's own credentials to log into carrier portals. This means the platform must securely store and manage credentials for every carrier portal for every agency it serves. If an agency has appointments with 20 carriers, the platform needs to manage 20 sets of credentials per agency, handle password changes, manage multi-factor authentication, and ensure that credential usage patterns don't trigger carrier security systems.

Error Recovery

In manual quoting, when something goes wrong — a page doesn't load, a validation error appears, a session times out — the agent sees the problem and adapts. Automated systems must handle the same range of errors programmatically, deciding when to retry, when to use a different approach, and when to escalate to a human.

The error taxonomy for carrier portal automation includes:

• Transient errors: Page timeouts, slow loading, temporary server issues — typically resolved by retry
• Validation errors: Carrier-specific data requirements that weren't anticipated — requires data transformation or user input
• Portal changes: UI modifications that break the automation — requires integration update
• Business logic errors: The carrier declines the risk, requires additional information, or routes to manual underwriting — requires communication back to the agent
• Authentication errors: Credential issues, MFA challenges, account lockouts — requires credential management

A production system must handle all of these gracefully, without losing the agent's data or requiring them to start over.

Portal Change Velocity

Carrier portals change frequently. Major redesigns happen annually or bi-annually. Minor changes — field additions, dropdown value changes, workflow reordering — happen quarterly or more often. A platform maintaining integrations with 40 to 50 carrier portals can expect several changes per week across its carrier panel.

The speed at which a platform detects and adapts to portal changes is a critical quality metric. A system that takes a week to adapt to a portal change has a week of downtime for that carrier. A system that detects changes within hours and adapts within a day provides near-continuous coverage.

The Infrastructure Argument

The core thesis of this analysis is that AI web agents is transitioning from a product feature to an infrastructure layer. This distinction matters because it affects how the technology is built, valued, and adopted.

Feature vs. Infrastructure

When AI web agents is a feature, it's a capability within a specific product. QuoteSweep's AI web agents is a feature of QuoteSweep. Semsee's API connections are a feature of Semsee. Each product builds and maintains its own carrier connectivity layer.

When AI web agents becomes infrastructure, it's a shared layer that multiple products build on top of. Just as Stripe handles payment processing for thousands of e-commerce applications (none of which build their own payment infrastructure), a AI web agents infrastructure layer would handle carrier connectivity for multiple insurance technology products.

The Fintech Parallel in Detail

The evolution from feature to infrastructure has played out in fintech over the past decade:

Phase 1: Screen-scraping as a feature (2010-2015). Early fintech apps built their own bank connectivity through screen-scraping. Each app maintained its own integrations. This was expensive, brittle, and duplicative.

Phase 2: Consolidation into infrastructure (2015-2020). Plaid, Yodlee, and others consolidated bank connectivity into an infrastructure layer. Fintech apps stopped building their own bank integrations and started using these platforms. This reduced cost, improved reliability, and accelerated the pace of fintech innovation.

Phase 3: API migration (2020-present). As banks adopted open banking standards and built APIs, the infrastructure providers migrated connections from screen-scraping to APIs where available, maintaining screen-scraping for institutions without APIs. The transition was invisible to the fintech apps built on top of the infrastructure.

Insurance is in Phase 1, with early signs of Phase 2. Multiple insurance technology companies are building and maintaining their own carrier connectivity — whether through APIs, AI web agents, or both. This duplicated effort represents an industry-wide inefficiency.

What Insurance Infrastructure Looks Like

An insurance carrier connectivity infrastructure layer would:

1. Maintain carrier portal integrations across the broadest possible set of carriers, handling portal changes, CAPTCHA challenges, and authentication
2. Offer standardized APIs that insurance technology products can build on — submit an application, get a quote, check appetite, retrieve status
3. Handle credential management securely, so individual products don't need to build their own credential storage
4. Absorb portal change maintenance so individual products aren't each independently adapting to the same portal change
5. Migrate to carrier APIs where available, maintaining AI web agents where APIs don't exist — transparent to the products built on top

This is precisely the pattern that Plaid followed for financial institution connectivity.

Why It Hasn't Happened Yet

Several factors have prevented the emergence of a dominant carrier connectivity infrastructure provider:

The market is still maturing. Commercial insurance technology is a smaller market than fintech, with fewer venture-funded companies competing for connectivity. The pressure to consolidate connectivity into infrastructure hasn't reached critical mass.

Carrier relationships are complex. Using AI web agents to interact with a carrier's portal exists in a gray area of the carrier's terms of service. Infrastructure providers need carrier relationships to operate sustainably, and building those relationships takes time.

Each product has different needs. A comparative rater needs quoting connectivity. An AMS needs policy download. A certificate management tool needs certificate issuance. The data models and workflows are different enough that a single infrastructure layer is technically complex.

Competitive dynamics. Companies that have invested in building their own carrier connectivity view it as a competitive moat. Sharing that connectivity as infrastructure would reduce their differentiation. This is a common tension in the feature-to-infrastructure transition — the same tension that early fintech companies felt about sharing bank connectivity.

QuoteSweep's Position in This Landscape

QuoteSweep has built its own carrier connectivity layer powered by AI web agents, with API connections where available. The platform's AI web agents interact with carrier portals directly, supported by field mapping intelligence that translates standardized application data into each carrier's specific portal format.

This connectivity layer enables real-time quoting across a broad carrier panel — including regional and specialty carriers that have no API. The appetite checking layer filters out carriers that won't write a given risk before initiating any portal interactions, avoiding wasted automation cycles.

Whether the insurance industry converges on a shared connectivity infrastructure layer or continues with each platform maintaining its own is an open question. What's clear is that AI web agents — in whatever organizational form — is the mechanism through which the majority of carrier connectivity will operate for the foreseeable future.

What This Means for Different Stakeholders

For Insurance Agents

The practical implication is straightforward: the tools you use to quote commercial insurance will increasingly be powered by AI web agents, whether or not the tool markets itself that way. When a comparative rater quotes 15 carriers in 5 minutes, the technology making that possible is AI web agents for the majority of those carriers.

Agents should evaluate quoting tools based on carrier coverage breadth (how many of your appointed carriers are supported), reliability (what percentage of submissions complete successfully), and adaptation speed (how quickly does the tool recover from portal changes). These are infrastructure quality metrics.

For Carriers

Carrier portals are being automated whether carriers plan for it or not. This creates a choice: carriers can ignore AI web agents (and accept that their portal experience is mediated by third-party tools), resist it (which may impair agent productivity and competitiveness), or embrace it (by making portals more automation-friendly or by building APIs).

The carriers that benefit most from this trend are those with the broadest agent distribution — their portals are the ones being automated first, and agents using automation tools are submitting more business to those carriers.

For Insurance Technology Companies

The build-vs-buy decision for carrier connectivity is becoming more consequential. Building and maintaining your own carrier portal integrations is expensive and requires specialized engineering talent. As the market matures, the economic argument for shared infrastructure becomes stronger.

For Investors and Analysts

AI web agents in insurance is infrastructure, not a feature. This has valuation implications. Infrastructure businesses (Stripe, Plaid, Twilio) are valued differently from feature businesses because infrastructure has network effects, switching costs, and compounding value as more products build on top of it. The insurance technology companies that are building carrier connectivity may be building infrastructure — even if they don't describe it that way today.

The Five-Year View

Projecting forward five years, several trends are likely:

AI web agents reliability will continue to improve. The combination of better monitoring, AI-assisted adaptation, and accumulated operational experience will push reliability above 99% for well-maintained integrations.

AI-driven browser agents will become production-grade. Current limitations in reliability and accuracy will be resolved through improved models, better guardrails, and hybrid architectures. This will reduce the per-carrier integration cost and enable faster expansion to new portals.

Some consolidation into infrastructure will occur. At least one company will emerge as a shared connectivity layer for insurance, analogous to Plaid's role in fintech. Whether this comes from an existing carrier connectivity player or a new entrant is uncertain.

Carrier API adoption will increase marginally. More carriers will build APIs, but the adoption rate will remain slow relative to the total carrier count. AI web agents and APIs will coexist indefinitely, with automation covering the long tail.

The connectivity layer will expand beyond quoting. Early AI web agents focuses on quoting. Infrastructure maturity will extend to binding, policy issuance, endorsements, audits, and claims — the full policy lifecycle.

The insurance industry's carrier connectivity problem is a temporary infrastructure gap, not a permanent condition. The question is not whether the gap will be bridged, but how — and AI web agents is the bridge.

Frequently Asked Questions

How is AI web agents different from RPA in insurance?

Traditional RPA (robotic process automation) records and replays human interactions, simulating mouse movements and keystrokes. It tends to be brittle, breaking when carrier portals change their layout. Modern AI web agents using headless frameworks like Playwright interacts with web pages at the code level (the DOM), making it faster, more precise, and more resilient to portal changes. In practice, most production insurance automation systems use headless browser technology rather than traditional RPA.

Is AI web agents legal for insurance carrier portals?

AI web agents using the agent's own credentials performs the same actions the agent would perform manually — logging in, filling forms, submitting applications. The technology itself is a method of interaction, not a circumvention of access controls. However, carrier portal terms of service vary, and the relationship between automation platforms and carriers is evolving. The strongest automation platforms maintain direct carrier relationships and operate with carrier awareness or explicit approval.

Will carrier APIs eventually make AI web agents obsolete?

Unlikely in the foreseeable future. Carrier API adoption has been slow for over a decade, and the structural barriers — cost, legacy systems, carrier IT priorities — haven't changed. Even optimistic projections suggest over 95% of carriers will remain portal-only for years to come. The more likely outcome is coexistence: APIs for the largest carriers and AI web agents for the rest. This mirrors what happened in fintech, where Plaid still maintains screen-scraping connections alongside bank APIs.

How does AI web agents handle carrier portal changes?

This is the primary ongoing challenge. Production-grade platforms use monitoring systems that detect portal changes (DOM diffing, visual regression testing, automated test suites) and adaptation processes that update integrations quickly. The best platforms detect changes within hours and adapt within a day. Less mature platforms may take days or weeks, resulting in periods where specific carriers are unavailable. Adaptation speed is one of the most important quality metrics for evaluating AI web agents platforms.

What carriers does AI web agents work with?

AI web agents works with any carrier that has a web-based quoting portal — which includes the vast majority of commercial carriers. It does not require carrier cooperation or API access. This is its primary advantage over API-only connectivity: instead of being limited to the approximately 50 carriers with APIs, AI web agents can potentially reach any carrier in the market. The practical limitation is the engineering investment required to build and maintain each carrier integration.