If you run an e-commerce brand, retail pricing desk, or market intelligence pipeline, Google Shopping and Google News are two of the highest-signal public data sources on the internet. Google Shopping aggregates live SKU pricing, promotional discounts, shipping terms, and checkout links across Amazon, Walmart, Target, Best Buy, eBay, and thousands of direct-to-consumer (D2C) Shopify and WooCommerce stores in 80+ countries.
Yet developers who try to query Google's official APIs quickly hit a wall:
- Google Merchant API Only Reads Your Own Store: Google's Content API for Shopping (and the newer Merchant API) is strictly for managing your own Merchant Center catalog. It provides zero endpoints to search competitor products or compare multi-merchant checkout offers.
- Obfuscated Merchant Redirect Links & News URLs: On both Google Shopping and Google News (
news.google.com), Google wraps outbound seller and publisher links in tracking redirects or base64/protobuf tokensβforcing scrapers to unwrap links cleanly if you want the true destination domain (merchantDomain/publisherDomain) and direct URL. - High Cost of Legacy Retail & Media APIs: Enterprise SERP and news APIs routinely charge
$10β$35 per 1,000 requestsor require$250+/monthsubscriptions before returning full-article text or multi-seller comparison offers.Why Teams Combine Google Shopping (
$2.00/1K) & Google News ($1.00/1K): E-commerce pricing teams, D2C brands enforcing Minimum Advertised Price (MAP) policies, and quantitative analysts use Google Shopping Scraper (unitbytes/google-shopping-scraper) to track live SKU prices and multi-merchant offers across 80+ countries, and pair it with Google News Scraper (unitbytes/google-news-scraper) to monitor competitor product launches, recalls, supply-chain events, and PR coverage in real time.
1. UnitBytes Google Shopping & News Suite vs. Legacy APIs#
| Capability | Google Shopping Scraper ($2.00β$4.00/1K) | Google News Scraper ($1.00β$2.50/1K) | Legacy SERP / News APIs ($10β$35/1K) |
|---|---|---|---|
| Primary Data Surface | Google Shopping Product & Offer Matrix (80+ countries) | Google Search News + news.google.com RSS/Topics (35+ countries) | Single-surface SERP HTML |
| Direct Destination URLs | Unwrapped merchantUrl + merchantDomain | Direct unwrapped url + publisherDomain | Often stuck on google.com/url or news.google.com/rss/articles/... |
| Multi-Merchant Offer Matrix | offers[] array (merchantName, price, delivery, merchantUrl) | relatedArticles[] cluster | Usually top 1 offer only |
| Rich Item Metadata | price, oldPrice, discountBadge, isOnSale, rating, reviewsCount, specifications | title, snippet, publishedDateIso, author, optional full articleText & wordCount | Snippet-only without expensive add-ons |
| Pay-Per-Event Pricing | $0.002β$0.004 / product ($2.00β$4.00 / 1,000) | $0.001β$0.0025 / article ($1.00β$2.50 / 1,000) | $10.00β$35.00 / 1,000 |
With Apify's $5.00 free monthly platform credit, you can extract 1,250β2,500 Google Shopping products or 2,000β5,000 Google News articles every month for free.
2. Google Shopping Scraper: Complete Product & Multi-Seller Schema#
Every product returned by Google Shopping Scraper & API (unitbytes/google-shopping-scraper) is normalized into a 31-field JSON object that works out-of-the-box in Excel/CSV or relational databases:
{
"productId": "7530392338343106684",
"catalogId": "7530392338343106684",
"offerId": "1194028475102938471",
"gpcId": "492817364519283746",
"title": "Keychron K8 Tenkeyless Wireless Mechanical Keyboard",
"brand": "Keychron",
"price": 79.99,
"currency": "USD",
"priceFormatted": "$79.99",
"oldPrice": 99.99,
"oldPriceFormatted": "$99.99",
"discountBadge": "20% off",
"isOnSale": true,
"merchantName": "Keychron",
"merchantDomain": "keychron.com",
"merchantUrl": "https://www.keychron.com/products/keychron-k8-tenkeyless-wireless-mechanical-keyboard",
"hasMultipleOffers": true,
"offersCount": 4,
"offers": [
{
"merchantName": "Keychron",
"merchantDomain": "keychron.com",
"price": 79.99,
"currency": "USD",
"priceFormatted": "$79.99",
"delivery": "Free delivery",
"merchantUrl": "https://www.keychron.com/products/keychron-k8-tenkeyless-wireless-mechanical-keyboard"
},
{
"merchantName": "Amazon.com",
"merchantDomain": "amazon.com",
"price": 84.99,
"currency": "USD",
"priceFormatted": "$84.99",
"delivery": "Free delivery by Fri",
"merchantUrl": "https://www.amazon.com/dp/B08B5W1234"
}
],
"rating": 4.7,
"reviewsCount": 1326,
"delivery": "Free delivery",
"freeShipping": true,
"returnPolicy": "Free 30-day returns",
"condition": null,
"specifications": {
"Brand": "Keychron",
"Connectivity": "Bluetooth, USB-C",
"Backlit": "RGB"
},
"imageUrl": "https://encrypted-tbn2.gstatic.com/shopping?q=tbn:ANd9GcR_example_product_thumb&usqp=CAc",
"searchQuery": "wireless mechanical keyboard",
"position": 1,
"country": "us",
"scrapedAt": "2026-11-12T12:00:00Z"
}
Built-In Filtering for MAP Compliance & Deal Arbitrage
- Price Bounds (
minPrice/maxPrice): Filter out cheap accessories (like replacement keycaps or cables) when monitoring high-ticket hardware SKUs. - Sale & Shipping Filters (
onSaleOnly: true,freeShippingOnly: true): Isolate active promotional markdowns (isOnSale: true,oldPrice,discountBadge). - Condition Filter (
condition:"new","used","refurbished"): Separate authorized new-in-box retail inventory from secondary/refurbished sellers.
3. Google News Scraper: Direct Publisher URLs & Optional Full Article Text#
When you need real-time media monitoring, PR attribution, or clean news context for LLM/RAG pipelines, Google News Scraper & API (unitbytes/google-news-scraper) supports keyword queries, category topics (BUSINESS, TECHNOLOGY, FINANCE, SCIENCE, HEALTH), source domains filtering, exactPhrase, excludeWords, and custom timeRange (1h, 24h, 7d, 30d, 1y, or dateFrom/dateTo).
{
"articleId": "CPcBSvDhrPhUpM01",
"title": "Next-Generation Grid Storage Deployments Double Across North America",
"url": "https://cleanenergy-journal-example.com/2026/11/12/grid-storage-deployments-double",
"publisher": "CleanEnergy Journal",
"publisherDomain": "cleanenergy-journal-example.com",
"publisherUrl": "https://cleanenergy-journal-example.com",
"snippet": "Utility-scale battery installations expanded by 104% year-over-year as regional grid operators accelerated peak-load balancing programs...",
"publishedAt": "2026-11-12T12:15:00+00:00",
"publishedAtText": "2 hours ago",
"publishedDateIso": "2026-11-12T12:15:00+00:00",
"author": "Alex Rivera",
"articleText": "Utility-scale battery installations expanded by 104% year-over-year across North American interconnects...\n\nGrid planners cited declining lithium-iron-phosphate pack costs and streamlined interconnection queues...",
"wordCount": 642,
"sourceSurface": "google_search_news",
"searchQuery": "grid storage deployments",
"country": "us",
"language": "en",
"scrapedAt": "2026-11-12T14:20:05.112000+00:00"
}
Tip for Fast Headline Feeds vs. Deep RAG Ingestion: LeaveextractArticleContent: false(default) for ultra-fast headline, snippet, and unwrappedurlextraction across hundreds of articles in seconds, or setextractArticleContent: truewhen your LLM or sentiment pipeline needs the full body (articleText,author, andwordCount) fetched directly from the publisher page.
4. Python Quickstart: Automated MAP & Competitor Price Audit#
Install apify-client and pandas (pip install apify-client pandas) to check whether any retailer is undercutting your Minimum Advertised Price (MAP) across Google Shopping:
import os
import pandas as pd
from apify_client import ApifyClient
client = ApifyClient(os.environ["APIFY_TOKEN"])
MAP_FLOOR_USD = 85.00
run = client.actor("unitbytes/google-shopping-scraper").call(
run_input={
"queries": ["Keychron K8 Wireless Mechanical Keyboard"],
"country": "us",
"language": "en",
"condition": "new",
"minPrice": 40,
"maxItems": 40,
}
)
products = client.dataset(run["defaultDatasetId"]).list_items().items
violations = []
for p in products:
price = p.get("price")
if price is not None and price < MAP_FLOOR_USD:
violations.append({
"title": p.get("title"),
"price": price,
"oldPrice": p.get("oldPrice"),
"discountBadge": p.get("discountBadge"),
"merchantName": p.get("merchantName"),
"merchantDomain": p.get("merchantDomain"),
"merchantUrl": p.get("merchantUrl"),
"offersCount": p.get("offersCount"),
})
print(f"Scanned {len(products)} listings; found {len(violations)} offers below ${MAP_FLOOR_USD:.2f} MAP")
pd.DataFrame(violations).to_csv("google_shopping_map_violations.csv", index=False)
5. Frequently Asked Questions (FAQ)#
Does Google Shopping Scraper return direct retailer URLs or Google redirect links?
Every item includes the unwrapped merchantUrl and normalized merchantDomain (such as amazon.com, walmart.com, bestbuy.com, or a standalone Shopify store domain), alongside the googleShoppingUrl product comparison link.
How can I combine Google Shopping prices with search demand trends and Asian factory costs?
E-commerce sourcing teams frequently combine three UnitBytes engines in a single workflow:
- Google Shopping Trends Scraper (
$0.30/1K) to spot rising product search velocity before seasonal peaks. - Google Shopping Scraper (
$2.00/1K) to benchmark current US/EU retail selling prices (price), review counts (reviewsCount), and merchant competition (offersCount). - 1688.com Wholesale Factory Scraper (
$1.50/1K) or Alibaba B2B Scraper ($1.50/1K) to compare factory-floor MOQ unit costs against Western retail prices.
π Start Scraping Google Shopping & Google News Today#
Deploy on Apify with $5.00 in free monthly platform credits (zero login or Google API keys required):
- ποΈ Google Shopping Scraper & API ($2.00/1K β Live Prices, Multi-Merchant Offers & Checkout URLs)
- π° Google News Scraper & API ($1.00/1K β Direct Publisher URLs, Time Filters & Full Article Text)
- π Google Shopping Trends Scraper ($0.30/1K β Breakout Product Demand & Regional Interest)
Need a custom scheduled MAP compliance or 100,000-SKU retail price feed delivered to Snowflake, BigQuery, S3, or Excel? Request a quote via the Done-For-You Dataset Form below or email contact@unitbytes.com.
Monitor Google Shopping Prices ($2.00/1K) & Google News ($1.00/1K)
Extract live product prices, promotional markdowns, multi-merchant checkout links, and direct publisher news articles across 80+ countries with zero Google API keys.
Related Technical Guides & Teardowns
Scraping Google Jobs with Direct ATS Apply Links ($2.50/1K): The Python JobSpy & TheirStack Alternative
Technical comparison and Python implementation guide for extracting Google Jobs with resolved Greenhouse, Lever, Workday, and Ashby direct apply URLs, normalized salaries, and B2B hiring signals at $2.50/1Kβwithout python-jobspy HTTP 429 blocks or $20/1K enterprise job APIs.
Google Flights ($0.50/1K) & Google Hotels API in Python: The Post-Amadeus & fast-flights Alternative
Developer guide to querying real-time Google Flights airfare matrices ($0.50/1K Gold+, $0.90/1K Free) and Google Hotels multi-OTA rate parity across Booking.com, Expedia, and direct hotel sites ($2.50/1K Gold+) in Pythonβreplacing restricted GDS APIs and brittle HTML scrapers.
Google AI Mode & AI Overviews Brand Citation Audit in Python ($3β$4/1K vs. $495/mo GEO SaaS)
Developer playbook for building an automated Generative Engine Optimization (GEO) and Answer Engine Optimization (AEO) brand citation tracker across Google AI Overviews ($3.00/1K Gold+, $2.50/1K Free) and Google AI Mode ($4.00/1K Gold+) in Pythonβreplacing $495/month SaaS seats.