Methodology

Food Price Intelligence aggregates data from over 30 official government and international sources through automated API integrations. Our data pipeline runs on a scheduled basis, pulling the latest available data, normalizing it into a consistent format, and storing it in our database for analysis and display.

This page explains our data collection process, analytical methods, and the limitations of our approach. We believe in full transparency about how our data is sourced and processed.

Commodity prices are sourced primarily from the USDA Agricultural Marketing Service (AMS) Market News reports. These represent actual wholesale prices at U.S. terminal markets — the prices at which commodities change hands between distributors and retailers.

Each price record includes: commodity name, variety, market location, package size, low/high/average price, organic status, origin, and report date. Prices are stored as reported by USDA without modification. When multiple reports exist for the same commodity/variety/market/date combination, we deduplicate using unique composite keys.

What prices represent

These are wholesale terminal market prices, not retail (grocery store) prices. They reflect what buyers pay at distribution points and are leading indicators of retail price changes. Retail prices typically follow with a lag of 2-6 weeks.

Our price forecasts use statistical time series models trained on historical price data for each commodity. The forecasting pipeline:

1. Collects at least 90 days of historical daily average prices
2. Applies seasonal decomposition to separate trend, seasonal, and residual components
3. Fits an exponential smoothing model (ETS) or ARIMA model depending on data characteristics
4. Generates 30-day ahead forecasts with 80% and 95% confidence intervals
5. Runs anomaly detection to flag unusual price movements

Limitations

Forecasts are statistical projections based on historical patterns. They cannot predict sudden events such as weather disasters, trade policy changes, disease outbreaks, or geopolitical disruptions. Forecast accuracy varies by commodity and market conditions. Always use forecasts as one input among many in your decision-making process.

Each commodity receives a weather risk score (0-10) based on current conditions in its primary growing regions. The score combines:

• Drought severity — from U.S. Drought Monitor (D0-D4 scale)
• Active weather alerts — NWS warnings for agricultural areas
• Temperature anomalies — deviation from historical norms
• Precipitation patterns — excess or deficit vs. crop needs

The estimated price impact percentage is derived from historical correlations between similar weather conditions and subsequent price changes for each commodity. Risk levels are categorized as: low medium high.

We aggregate agricultural news from multiple sources via the GNews API and analyze sentiment using natural language processing. Each article receives:

• A sentiment score (-1.0 to +1.0) indicating bearish to bullish tone
• A sentiment label (positive, negative, or neutral)
• Commodity mentions — which commodities the article discusses
• A relevance score indicating how pertinent the article is to agricultural markets

Market sentiment is aggregated across recent articles to produce an overall market mood indicator (bullish, bearish, or neutral) displayed on the dashboard.

Several features use large language models (LLMs) to generate natural language analysis:

• Daily Briefing — AI-generated market summary combining price trends, weather risks, sentiment, and macro indicators
• AI Chat — conversational assistant that can query live market data and answer questions about commodity prices
• News Summarization — automated extraction of key insights from news articles

AI outputs are generated from real data but may contain inaccuracies. They are provided as supplementary analysis and should be verified against source data.

We take several steps to ensure data quality:

• Deduplication — unique composite keys prevent duplicate records
• Source verification — all data comes from official government APIs with established data standards
• Automated monitoring — ingestion jobs log errors and failed fetches for review
• Caching with TTL — cached computations expire automatically to ensure freshness

However, we cannot guarantee 100% accuracy. Government APIs occasionally report corrections or revised figures. If you notice a data discrepancy, please report it through our Feedback page.