Add MLS by HansonXyz plugin for MLS Grid API integration

Features:
- Full sync of NorthStar MLS properties via MLS Grid API v2
- Incremental sync using ModificationTimestamp
- Local media download and storage
- Rate limit compliance (2 req/sec, 7200/hr, 40000/day)
- Sync state tracking with resume capability
- WP-CLI commands: test, sync, status, stats, cache
- Admin settings page with manual sync triggers
- Public API functions: mls_get_properties, mls_get_property, etc.

Database tables:
- mls_properties: Listing data with full field mapping
- mls_media: Downloaded images
- mls_sync_state: Sync progress tracking
- mls_rate_limits: API usage tracking
- mls_sync_log: Debug logging

Documentation:
- docs/CLAUDE.md: AI development guide
- docs/API.md: MLS Grid API reference
- docs/USAGE.md: User documentation

Tested: Connection, auth, sync 10 records, media download verified

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Hanson.xyz Dev
2025-12-14 21:24:38 -06:00
parent ec5a309555
commit 6556479417
18 changed files with 5324 additions and 10 deletions
@@ -0,0 +1,298 @@
# MLS Grid API Reference
Documentation for the MLS Grid API v2 used by this plugin.
## Base URL
```
https://api.mlsgrid.com/v2
```
## Authentication
Bearer token authentication via HTTP header:
```
Authorization: Bearer {access_token}
```
**Required Header:**
```
Accept-Encoding: gzip
```
The API requires gzip compression and will return error 400 without it.
## Rate Limits
| Limit | Value |
|-------|-------|
| Per Second | 2 requests |
| Per Hour | 7,200 requests |
| Per Day | 40,000 requests |
| Data Per Hour | 4GB |
Exceeding limits returns HTTP 429 and temporarily suspends access.
## Endpoints
### Property
```
GET /Property
```
Main endpoint for listing data.
**Query Parameters:**
- `$filter` - OData filter expression (required)
- `$expand` - Include related resources: Media, Rooms, UnitTypes
- `$top` - Records per page (max 5000, max 1000 with $expand)
- `$select` - Specific fields to return
- `$orderby` - Sort order
**Example:**
```
/Property?$filter=OriginatingSystemName eq 'northstar' and MlgCanView eq true&$expand=Media&$top=1000
```
### Member
```
GET /Member
```
Agent/member records.
### Office
```
GET /Office
```
Brokerage office records.
### OpenHouse
```
GET /OpenHouse
```
Open house event records.
### Lookup
```
GET /Lookup
```
Field value definitions. Query no more than once per day.
## OData Filter Syntax
### Operators
| Operator | Description | Example |
|----------|-------------|---------|
| eq | Equals | `City eq 'Austin'` |
| ne | Not equals | `Status ne 'Sold'` |
| gt | Greater than | `ListPrice gt 200000` |
| ge | Greater or equal | `BedroomsTotal ge 3` |
| lt | Less than | `ListPrice lt 500000` |
| le | Less or equal | `YearBuilt le 2020` |
| and | Logical AND | `City eq 'Austin' and BedroomsTotal ge 3` |
| or | Logical OR | Limited to 5 per query |
| in | In list | `City in ('Austin', 'Dallas')` |
### Required Filters
Every Property request MUST include:
```
OriginatingSystemName eq 'northstar'
```
For initial import, add:
```
MlgCanView eq true
```
### Timestamp Filters
For incremental sync:
```
ModificationTimestamp gt 2024-01-15T00:00:00.000Z
```
## Pagination
Responses include `@odata.nextLink` field containing URL for next page.
```json
{
"@odata.context": "...",
"value": [...],
"@odata.nextLink": "https://api.mlsgrid.com/v2/Property?$filter=...&$skip=1000"
}
```
Continue fetching until `@odata.nextLink` is absent.
## Property Fields
### Core Fields
| Field | Type | Description |
|-------|------|-------------|
| ListingKey | string | Unique identifier |
| ListingId | string | MLS listing number |
| StandardStatus | string | Active, Pending, Closed, etc. |
| ListPrice | decimal | Listing price |
| ClosePrice | decimal | Sold price |
### Address Fields
| Field | Type | Description |
|-------|------|-------------|
| StreetNumber | string | Street number |
| StreetName | string | Street name |
| StreetSuffix | string | St, Ave, Blvd, etc. |
| UnitNumber | string | Unit/apt number |
| City | string | City name |
| StateOrProvince | string | State abbreviation |
| PostalCode | string | ZIP code |
| CountyOrParish | string | County name |
| Latitude | decimal | GPS latitude |
| Longitude | decimal | GPS longitude |
### Property Details
| Field | Type | Description |
|-------|------|-------------|
| PropertyType | string | Residential, Land, Commercial, etc. |
| PropertySubType | string | Single Family, Condo, etc. |
| BedroomsTotal | integer | Total bedrooms |
| BathroomsTotalInteger | integer | Total bathrooms |
| BathroomsFull | integer | Full bathrooms |
| BathroomsHalf | integer | Half bathrooms |
| LivingArea | integer | Square feet |
| LotSizeArea | decimal | Lot size |
| LotSizeUnits | string | Acres, SqFt |
| YearBuilt | integer | Year built |
| GarageSpaces | integer | Garage spaces |
### Description Fields
| Field | Type | Description |
|-------|------|-------------|
| PublicRemarks | string | Property description |
| Directions | string | Driving directions |
### Agent/Office Fields
| Field | Type | Description |
|-------|------|-------------|
| ListAgentKey | string | Listing agent ID |
| ListAgentMlsId | string | Agent MLS ID |
| ListOfficeKey | string | Listing office ID |
| ListOfficeName | string | Office name |
| ListOfficeMlsId | string | Office MLS ID |
### Timestamps
| Field | Type | Description |
|-------|------|-------------|
| ModificationTimestamp | datetime | Last modified (use for sync) |
| PhotosChangeTimestamp | datetime | Media last changed |
| ListingContractDate | date | Listed date |
| CloseDate | date | Sold date |
| DaysOnMarket | integer | DOM count |
### MLS Grid Fields
| Field | Type | Description |
|-------|------|-------------|
| MlgCanView | boolean | OK to display (false = delete) |
| MlgCanUse | array | Permitted use cases (IDX, VOW, etc.) |
| OriginatingSystemName | string | Source MLS identifier |
## Media (via $expand)
When using `$expand=Media`, each property includes Media array:
```json
{
"Media": [
{
"MediaKey": "abc123",
"MediaURL": "https://media.mlsgrid.com/...",
"Order": 1,
"ImageWidth": 1200,
"ImageHeight": 800,
"MediaModificationTimestamp": "2024-01-15T10:30:00Z"
}
]
}
```
**Important:** MediaURL is for downloading only. Store images locally.
## Sync Strategy
### Initial Import
1. Query with `MlgCanView eq true` to get viewable records
2. Follow `@odata.nextLink` for pagination
3. Store `ModificationTimestamp` from last record
### Incremental Sync
1. Query with `ModificationTimestamp gt {last_timestamp}`
2. Do NOT filter by MlgCanView (need to see deletions)
3. If `MlgCanView = false`, delete local record
### Media Sync
1. Check `PhotosChangeTimestamp` on each property
2. If changed, replace all media for that listing
3. Match by `MediaKey`, download via `MediaURL`
4. Delete media where `MediaKey` no longer exists
### Error Recovery
Store `@odata.nextLink` after each page. On failure, resume from that URL.
## Best Practices
1. **Sequential requests only** - Do not parallelize API calls
2. **Respect rate limits** - 2 req/sec max, pause if approaching limits
3. **Use $expand wisely** - Reduces per-page limit from 5000 to 1000
4. **Store raw JSON** - Keep original response for debugging
5. **Query Lookup sparingly** - Once per day maximum
6. **Don't hotlink media** - Download and serve from local storage
## Error Responses
```json
{
"error": {
"code": 400,
"message": "Error description",
"target": "misc",
"details": []
}
}
```
| Code | Meaning |
|------|---------|
| 400 | Bad request (check filters, missing gzip) |
| 401 | Unauthorized (invalid token) |
| 429 | Rate limited (wait and retry) |
| 500+ | Server error (retry with backoff) |
## Resources
- [MLS Grid Documentation](https://docs.mlsgrid.com/)
- [API v2 Reference](https://docs.mlsgrid.com/api-documentation/api-version-2.0)
- [Best Practices Guide](https://www.mlsgrid.com/resources)
@@ -0,0 +1,140 @@
# MLS by HansonXyz Plugin
WordPress plugin for syncing MLS Grid API data (NorthStar MLS) into local database.
## Development Rules
1. **No emojis** - nowhere in code, commits, docs, or conversation
2. **PHP 7.4+** compatible code
3. **WordPress Coding Standards**
4. Follow patterns from existing HomeProz theme
## Quick Reference
### Database Tables
All tables use `{$wpdb->prefix}mls_` prefix:
| Table | Purpose |
|-------|---------|
| `mls_properties` | Listing data |
| `mls_media` | Media files |
| `mls_sync_state` | Sync progress tracking |
| `mls_rate_limits` | API usage tracking |
| `mls_sync_log` | Debug logging |
### API Configuration
Credentials in wp-config.php:
```php
define('MLSGRID_API_URL', 'https://api.mlsgrid.com/v2');
define('MLSGRID_ACCESS_TOKEN', 'your-token-here');
```
### MLS Grid API Rate Limits
MUST comply with these limits:
- 2 requests/second
- 7,200 requests/hour
- 40,000 requests/day
- 4GB data/hour
### Key Files
| File | Purpose |
|------|---------|
| `includes/class-mls-api-client.php` | API communication, auth, gzip |
| `includes/class-mls-sync-engine.php` | Sync orchestration |
| `includes/class-mls-media-handler.php` | Media download/storage |
| `includes/class-mls-query.php` | Public query API |
| `includes/class-mls-rate-limiter.php` | Rate limit compliance |
| `cli/class-mls-cli.php` | WP-CLI commands |
### WP-CLI Commands
```bash
# Test connectivity
wp mls test connection
wp mls test auth
# Show status
wp mls status
wp mls status rate-limits
# Run sync
wp mls sync full [--dry-run] [--limit=N]
wp mls sync incremental [--dry-run]
wp mls sync media [--limit=N]
wp mls sync resume --id=<sync_id>
# Statistics
wp mls stats
# Cache management
wp mls cache clear --confirm
wp mls cache cleanup
```
### Public API Functions
Available for themes/plugins:
```php
// Get properties with filters
$properties = mls_get_properties([
'status' => 'Active',
'city' => 'Albert Lea',
'min_price' => 100000,
'limit' => 20,
]);
// Get single property
$property = mls_get_property('NST123456');
// Get media
$media = mls_get_property_media('NST123456');
$image_url = mls_get_property_image('NST123456');
// Get distinct values
$cities = mls_get_cities('Active');
// Check data availability
if (mls_is_available()) { ... }
```
### Sync Strategy
1. **Initial Import**: Full sync downloads all viewable properties
2. **Incremental**: Uses ModificationTimestamp to fetch only changes
3. **Delete Handling**: MlgCanView=false triggers local deletion
4. **Media**: Downloads to wp-content/uploads/mls-listings/
5. **Recovery**: Stores last_next_link for resume on failure
### Testing After Changes
```bash
wp mls test connection
wp mls test auth
wp mls sync full --dry-run --limit=10
wp mls stats
```
### Property Data Mapping
Key fields from API to database:
| API Field | DB Column |
|-----------|-----------|
| ListingKey | listing_key |
| ListingId | listing_id |
| ListPrice | list_price |
| StandardStatus | standard_status |
| BedroomsTotal | bedrooms_total |
| BathroomsTotalInteger | bathrooms_total |
| LivingArea | living_area |
| City | city |
| ModificationTimestamp | modification_timestamp |
| PhotosChangeTimestamp | photos_change_timestamp |
| MlgCanView | mlg_can_view |
Full API response stored in `raw_data` column as JSON.
@@ -0,0 +1,244 @@
# MLS by HansonXyz - User Documentation
## Overview
This plugin syncs property listing data from MLS Grid (NorthStar MLS) into your WordPress database, making it available for use by themes and other plugins.
## Installation
1. Upload the `mls-by-hansonxyz` folder to `/wp-content/plugins/`
2. Activate the plugin through the WordPress admin
3. Configure API credentials (see below)
4. Run initial sync
## Configuration
### API Credentials
Add to your `wp-config.php`:
```php
define('MLSGRID_API_URL', 'https://api.mlsgrid.com/v2');
define('MLSGRID_ACCESS_TOKEN', 'your-access-token-here');
```
Alternatively, configure via Settings > MLS Settings in WordPress admin.
### Settings
Navigate to **Settings > MLS Settings** to configure:
- **Originating System**: MLS identifier (default: `northstar`)
- **Auto Sync**: Enable automatic background sync
- **Sync Interval**: How often to sync (30min to daily)
- **Sync Media**: Whether to download listing photos
## Running Sync
### Via Admin Panel
1. Go to Settings > MLS Settings
2. Click "Run Incremental Sync" or "Run Full Sync"
3. Wait for completion
### Via WP-CLI
```bash
# Test connection first
wp mls test connection
wp mls test auth
# Run initial full sync
wp mls sync full
# Run incremental updates
wp mls sync incremental
# Download pending media
wp mls sync media
```
### Via Cron
Add to your system crontab for scheduled sync:
```bash
# Run incremental sync every hour
0 * * * * cd /var/www/html && wp mls sync incremental --allow-root
```
## Checking Status
### Via Admin
Settings > MLS Settings shows:
- Database statistics (property counts by status)
- Last sync time and results
- Rate limit usage
### Via CLI
```bash
# Full status
wp mls status
# Just rate limits
wp mls status rate-limits
# Database statistics
wp mls stats
```
## Using the Data
### For Theme Developers
The plugin provides global helper functions:
```php
// Get active properties in a city
$properties = mls_get_properties([
'status' => 'Active',
'city' => 'Albert Lea',
'limit' => 20,
]);
foreach ($properties as $property) {
echo $property->list_price;
echo $property->bedrooms_total;
echo $property->city;
}
// Get a single property
$property = mls_get_property('NST123456');
// Get property images
$media = mls_get_property_media($property->listing_key);
$primary_image = mls_get_property_image($property->listing_key);
// Get cities with active listings
$cities = mls_get_cities('Active');
// Check if data is available
if (mls_is_available()) {
// Show property search
}
```
### Query Parameters
`mls_get_properties()` accepts these filters:
| Parameter | Type | Description |
|-----------|------|-------------|
| status | string | Active, Pending, Closed |
| property_type | string | Residential, Land, etc. |
| city | string | City name |
| county | string | County name |
| postal_code | string | ZIP code |
| min_price | int | Minimum price |
| max_price | int | Maximum price |
| min_beds | int | Minimum bedrooms |
| max_beds | int | Maximum bedrooms |
| min_baths | int | Minimum bathrooms |
| min_sqft | int | Minimum square feet |
| max_sqft | int | Maximum square feet |
| search | string | Search address/remarks |
| limit | int | Results per page |
| offset | int | Pagination offset |
| orderby | string | Sort field |
| order | string | ASC or DESC |
| include_media | bool | Include media array |
### Property Object Fields
Each property object includes:
```php
$property->listing_key // Unique ID
$property->listing_id // MLS number
$property->list_price // Price
$property->standard_status // Active, Pending, Closed
$property->street_number
$property->street_name
$property->street_suffix
$property->city
$property->state_or_province
$property->postal_code
$property->county
$property->latitude
$property->longitude
$property->property_type
$property->property_sub_type
$property->bedrooms_total
$property->bathrooms_total
$property->living_area // Square feet
$property->lot_size_area
$property->year_built
$property->garage_spaces
$property->public_remarks // Description
$property->directions
$property->list_office_name
$property->photos_count
$property->days_on_market
$property->modification_timestamp
```
## Media Storage
Downloaded images are stored in:
```
wp-content/uploads/mls-listings/{prefix}/{listing_key}/
```
Images are named by order: `1.jpg`, `2.jpg`, etc.
Access via:
```php
$media = mls_get_property_media($listing_key);
foreach ($media as $image) {
echo '<img src="' . esc_url($image->local_url) . '">';
}
```
## Troubleshooting
### Connection Failed
1. Verify API token is correct in wp-config.php
2. Check that MLSGRID_API_URL is set
3. Run `wp mls test connection` for details
### No Data After Sync
1. Check `wp mls status` for errors
2. Review rate limits - may need to wait
3. Check WordPress debug log for API errors
### Media Not Downloading
1. Verify `sync_media` is enabled in settings
2. Check upload directory is writable
3. Run `wp mls sync media` manually
### Rate Limit Exceeded
The plugin automatically waits when approaching limits. If suspended:
1. Wait for the rate limit window to reset
2. Reduce sync frequency
3. Contact MLS Grid support if persistent
### Clearing Data
To start fresh:
```bash
wp mls cache clear --confirm
```
This removes all synced data but keeps settings.
## Support
For plugin issues: Check logs at Settings > MLS Settings
For API issues: Contact MLS Grid support at support@mlsgrid.com