AutoGMap

AutoGMap — Business Discovery & WhatsApp Outreach

Discovers Indian businesses on Google Maps, enriches them with contact details, and runs WhatsApp campaigns to them through the Meta Cloud API — from a single dashboard.

┌──────────────────────────── React Dashboard ───────────────────────────┐
│  Discovery  │  Business Data  │  Campaign Builder  │  Campaign History │
└────────────────────────────────┬───────────────────────────────────────┘
                                 │  REST
                        ┌────────▼────────┐
                        │  FastAPI        │
                        └────┬───────┬────┘
              Playwright ◄───┘       └───► Meta WhatsApp Cloud API
             (Google Maps)                 (templates · sends · media)
                        ┌─────────────────┐
                        │   PostgreSQL    │  businesses · jobs · campaigns
                        └─────────────────┘
                                 ▲
                        n8n scheduled workflow

What it does

Discovery

• Searches Google Maps per (pincode × category) pair and extracts name, address, phone and website from each listing's detail page.
• Radius filtering. Google treats the map viewport as a hint, not a constraint, and regularly returns results hundreds of km away. Anything outside the location's radius is discarded rather than stored as a name-only row.
• Email enrichment crawls each business website for a contact address, on a strict time budget so a slow site cannot hold up a batch.
• Deduplication on place ID, name, phone and coordinates, so the same shop discovered under several categories is stored once.
• CAPTCHA is detected and the batch stops early rather than escalating.

Scraping autopilot

Dashboard → Scraping autopilot runs discovery on its own. Switch it on and it works in rounds:

• A round is one state and 5 batches of 25 jobs. Each batch is a different tehsil — in a different district while the state has unused ones — and no category is repeated within the round.
• Batches run one at a time, 15 minutes apart, whether or not a round ends between them. The next round is in a different state, the one visited longest ago.
• It stops for the day at 10 batches (IST) and carries on after midnight, picking up a round left part-done. A 25-job batch takes 15–20 minutes, so a day's ten take about five and a quarter hours.
• Only a batch that scraped something counts. One whose jobs all failed is not one of the day's ten. So that this cannot run without end, ten wholly failed batches in a day stop it until midnight with a note to look at why.
• Before each round it runs job generation, so newly added PINs join the queue.

To stay clear of Google's CAPTCHA it paces itself like a person, not a machine: randomised pauses between jobs, batches and rounds; a browser that reports its real version with an Indian locale. If a CAPTCHA appears anyway, everything stops for a cool-down that grows each time it recurs (30 min, 1 h, 2 h, 4 h), the pace slows afterwards, and the blocked job goes back in the queue. Three clean batches reset it. It never tries to solve a CAPTCHA.

Every setting — batches per day, batch size, gaps, which states — is under Pacing. Stop on a running batch also switches the autopilot off. It never runs alongside a batch started by hand, and after a restart it resumes the round it was in.

Failed jobs are tried again. Before each round, failed jobs go back in the queue and are scraped in the ordinary batches — the same ten a day, fifteen minutes apart, with no extra ones. Retries go first: a batch takes the jobs being tried again before ones never tried, so a failure is not left waiting behind thousands of untouched jobs. A job that failed because the internet was down goes back with that attempt not counted. Any other failure is retried until the job has used its attempts (JOB_RETRY_LIMIT, 3 by default), then it stays failed. Try failed jobs again under Pacing turns this off.

It waits out a dead internet connection. The connection is checked before every batch, so nothing is started into an outage. A batch that loses it part-way stops after three failed jobs in a row and puts them back in the queue, with the attempt not counted against them. The autopilot then checks again after 5, 15, 30 and then every 60 minutes, and carries on by itself when the connection is back. None of this uses up the day's batches. (On 3 October 2026 the server's link dropped and it ran 30 batches a day into it for two days, failing 1,490 jobs.)

It cannot take the server down. The machine is shared with other applications, so three things bound what scraping can use:

• Every browser a job starts is recorded against it. When the job ends — finished, or timed out and abandoned — anything it left running is killed. A wedged job used to leave its Chromium alive, a few hundred megabytes each.
• The backend container has a memory ceiling (BACKEND_MEMORY_LIMIT, 4 GB by default, swap included). Past it the kernel kills a process inside the container; at worst this container restarts, and nothing else is touched.
• A change of pace in the code is adopted once at the next start and leaves the autopilot switched off. A deploy never starts it; a person does.

Target locations

• The PIN list is editable. The spreadsheets seed it, but new PIN codes and towns are added from Configuration → Add PIN as the business expands into new geography, without touching the source files.
• Look up resolves a place through Google Maps and fills in the coordinates, and the state and PIN where Maps names them.
• State and district are required. Every business discovered at a location copies its state, district and tehsil, so a PIN saved without them yields rows that can never be filtered by geography. Maps states the state for a PIN code search but never the district, so that one is always typed rather than guessed.
• A PIN with jobs against it cannot be deleted, so businesses already discovered there are never orphaned.

Who can sign in

Access is two gates, not one.

1. The domain decides who may ask. Only @eko.co.in addresses can submit a signup request; anything else is refused at the API.
2. An admin decides who gets in. A signup creates a PENDING account that cannot sign in. It works only once an admin approves it.

The second gate is the point: the address proves the person is a colleague, not that this particular colleague should see campaign data and send messages from the company's WhatsApp number.

mansi.kanchan.intern@eko.co.in is seeded as the first admin at startup — otherwise the approvals queue would have nobody able to read it. That account gets its first password from BOOTSTRAP_ADMIN_PASSWORD, applied at startup only while it has none; an existing password is never overwritten. The signup form cannot set it: signup checks that an address is on the domain, not that the person typing it owns the mailbox, so whoever submitted it first would otherwise become admin.

Roles. Someone asking for access chooses the role they need; the administrator sees what was asked for and can give a different one when approving, or change it later from the same list.

Member is what a request gets unless more is asked for or given, so the least access is the default. A member's limits are enforced by the server on every request, not by hiding buttons: anything that is not a read is refused with "You have view-only access", whichever route it goes through. The page hides the tabs and controls a member cannot use so that it never offers them.

Remove access on an approved account takes effect on that person's next request, since every request looks the account up afresh; Restore access undoes it. The administrator's role cannot be granted, asked for, changed or removed from the app, so the approvals queue stays with that one account.

Passwords. Nobody can read a password: only a one-way hash is stored.

• Change password, in the account menu once signed in, asks for the current password and the new one.
• Forgot password?, on the sign-in page, sends a 6-digit code to the administrator, not to the person asking. The email names the account it is for; the administrator passes the code on if the request is genuine, and the person enters it with a new password. The code works once, for 10 minutes, and is thrown away after 5 wrong guesses; another is not sent for 60 seconds. Needs the SMTP settings. The page answers the same whether or not the address has an account.
• Either way, changing a password ends every session that was signed in with the old one.

Admins see Access Requests in the sidebar: approve, decline with a reason, or reinstate someone previously declined. Every decision records who made it. Requests also go out by email when SMTP is configured; when it is not, the queue is still the source of truth and signup is unaffected.

Everything under /api/ requires a session, enforced as middleware so a route added later is protected by default. The exceptions are the endpoints that cannot carry a cookie: /health, /ready, /api/v1/auth/*, the campaign redirect /r/{token}, and the Meta webhook — which is guarded by its signature instead.

Deploying behind a reverse proxy

Set VITE_BASE_PATH (with trailing slash) to the public sub-path before building — Vite rewrites every asset URL with it, so it cannot be changed afterwards. PUBLIC_BASE_URL, BACKEND_HOST_PORT and POSTGRES_HOST_PORT are runtime settings; the two ports bind to 127.0.0.1 only.

The media directory needs the container's uid. ./data is bind-mounted, so the host's ownership wins over the image's. The app runs as appuser (uid 996) and cannot write into a directory owned by whoever cloned the repo, which surfaces as "Failed to upload media" when saving a template with an image header:

sudo chown -R 996:996 data

Run it once after cloning, before starting the stack.

Turning the webhook on

Everything below the line is built and tested; these are the four steps that connect it to Meta.

1. Expose the backend over HTTPS and set PUBLIC_BASE_URL to it (e.g. https://link.eko.in). The same URL serves the click-tracking redirect, so this one variable switches both on.
2. Set META_APP_SECRET to the app secret from the Meta app dashboard. Until it is set, POST /api/v1/whatsapp/webhook answers 403 to every call, including Meta's — the signature cannot be verified without it.
3. Register the callback in Meta → WhatsApp → Configuration:
   • Callback URL: <PUBLIC_BASE_URL>/api/v1/whatsapp/webhook
   • Verify token: the value of META_WEBHOOK_VERIFY_TOKEN
4. Subscribe the fields messages (delivery statuses and quick-reply taps both arrive under it) and, optionally, message_template_status_update for template approvals.

Then send one message to your own number and confirm the row in Campaign History → View Details turns Delivered, then Read.

Note that Meta does not replay events for messages already sent, so campaigns run before this is switched on will never show delivery data.

The live Google Sheet

Business Data → Export holds CSV, Excel and one shared spreadsheet named All Scraped Data from AutoGMap.

Unlike the downloads, the sheet is a single document that is reused and rewritten rather than a new export each time, so a bookmarked link keeps working. "Update the sheet now" replaces its contents with the current table — appending would duplicate every row, and there is no key in the sheet to merge against.

Two settings are needed before it can exist:

The second is not optional in practice: a service account owns what it creates, so without an explicit share the sheet exists and nobody — including you — can open it. The sync endpoint refuses to run until it is set.

Access is Google's, not this app's. The sheet is shared with that one address; anyone else following the link gets Google's request-access page. This app never sees who opens it and cannot grant or revoke anything.

Delivery & engagement tracking

• Campaign History shows delivered, read and button clicks per campaign; View Details adds a Delivery & Engagement panel and per-recipient Delivered / Read / Clicked columns.
• Stages are timestamps, not one status. A message that was read was also delivered, so the stages accumulate. Collapsed into a single status field, READ overwrites DELIVERED and the delivered count is lost for good.
• A dash is not a zero. A sent message Meta has not reported on is unknown, not undelivered — only messages Meta actually failed are counted as such.
• Quick-reply taps are attributed through the context.id on the inbound message. Meta sends no event at all when a call-to-action URL button is tapped, so those are measured by the tracking link instead.
• Events arrive out of order and are repeated until acknowledged, so handling is monotonic and idempotent: a late "delivered" never undoes a "read", and a redelivered tap is never counted twice.
• Requires the webhook. Data only flows once /api/v1/whatsapp/webhook is reachable at a public HTTPS URL and subscribed to messages in the Meta app. Meta does not replay events for messages already sent, so campaigns run before that show no delivery data.

WhatsApp campaigns

• Template lifecycle against Meta. Templates are composed in the UI with a live preview, submitted for review automatically, and their status and billing category are synced back from the WhatsApp Business Account.
• Pre-flight checks. Before a campaign sends anything it verifies the template is APPROVED in the WABA and that its header media exists and fits Meta's per-type size limit. A misconfigured campaign fails once with a readable reason instead of one rejected recipient at a time.
• Contact validation normalises Indian numbers to +91XXXXXXXXXX, separating landlines, malformed entries and duplicates before sending.
• Resumable campaigns. A campaign stopped mid-run can continue for the recipients it never reached, without re-contacting anyone already sent to.
• Per-campaign execution log with Meta's own error codes, visible in the dashboard rather than only in container logs.

Inbox and the onboarding bot

Inbox lists everyone who has written to the business number: a typed message, a quick-reply tap, a photo. Each thread also shows the campaign messages that reached that number, so a reply is read under what it answers. A typed reply can be sent only within 24 hours of the person's last message, which is Meta's rule; after that the box says why it is closed instead.

The onboarding bot starts when someone taps a trigger button on a campaign (default "Interested", set under Inbox → Bot):

1. It asks for their PIN code.
2. The PIN is served if it is on our location list or within ONBOARDING_RADIUS_KM (20 km) of one. A PIN not on the list is placed on the map through OpenStreetMap, falling back to India Post; only the PIN is sent. Outside the area, they are told so politely.
3. "Can you invest ₹5 Lakh?" — Yes / No.
4. The documents needed, and "Do you have them?" — Yes / No.
5. A link to the upload page, /onboard/{token}, which is public, Eko branded and lapses after 7 days. Files are checked by their contents (PDF, JPG or PNG, up to 5 MB each).
6. On submit the page returns them to the WhatsApp chat, the bot thanks them, and the admin is emailed.

It answers only what it asked; anything else waits for a person. Pause the bot here in a thread hands that conversation to whoever is replying, and the switch under Inbox → Bot turns it off everywhere.

Documents go to Google Drive, one folder per person, inside GOOGLE_DRIVE_FOLDER_ID, using the same service account as the live sheet. The folder must be in a shared drive with that service account as Content manager: a service account has no storage of its own, so a folder in someone's My Drive is refused. Once Drive holds a file the server copy is deleted. If the copy fails the file stays on the server, the reason is shown in the thread, and Try again retries it.

Dashboard

• Light and dark themes, remembered per browser.
• Campaign history with a detail view: delivery counters, failure reasons grouped by cause, the template that was sent, and the full execution log.
• Timestamps render in the viewer's own timezone.

Quick Start — Local Development

Prerequisites

• Python 3.12+
• Node.js 20+
• Git

1. Clone and set up environment

git clone https://github.com/mansikanchan2003/G-Map-Data-Scraper.git
cd G-Map-Data-Scraper
cp .env.example .env
# Edit .env if needed — SQLite defaults work for local dev

2. Install Python dependencies

pip install -r requirements.txt

3. Install Playwright Chromium browser

python -m playwright install chromium
python -m playwright install-deps chromium   # Linux only

4. Initialize the database (SQLite — local dev)

# Option A: let the app create it on startup (automatic)
uvicorn src.main:app --reload

# Option B: use Alembic explicitly
alembic upgrade head
uvicorn src.main:app --reload

5. Verify the backend

curl http://localhost:8000/health
# Expected: {"status": "healthy", "database": "healthy", ...}

6. Install and start the frontend

cd frontend
npm install
npm run dev
# Open http://localhost:5173

Production Deployment

Prerequisites

• Linux server (Ubuntu 22.04+ or Debian 12+ recommended)
• Docker 24+ and Docker Compose v2
• PostgreSQL 16+ (or use the provided docker-compose)
• Minimum 2 GB RAM (Playwright Chromium requires ~500 MB per browser context)

Environment Variables

Copy .env.example to .env and configure all required values:

WhatsApp (Meta Cloud API)