Integrate MLS listings with property map and add smart sync
Property Map: - Replace ACF-based property display with MLS database queries - Use real lat/lng coordinates from MLS (100% coverage) - Create property-card-mls.php template for MLS property cards - Update AJAX handler to filter MLS properties MLS Plugin Enhancements: - Add 'wp mls run' smart sync command (auto-detects full/incremental/resume) - Add database index migrations for lat/lng and composite search indexes - Add comprehensive README.md documentation Documentation: - Update site README.md with sysadmin quick reference - Add FEATURES_PENDING_12_15.md tracking client feature requests 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Hanson.xyz Dev
@@ -0,0 +1,637 @@
|
||||
# MLS by HansonXyz
|
||||
|
||||
WordPress plugin for syncing MLS Grid API data (NorthStar MLS) into a local database with WP-CLI tools and a public API for themes and plugins.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Features](#features)
|
||||
- [Requirements](#requirements)
|
||||
- [Installation](#installation)
|
||||
- [Configuration](#configuration)
|
||||
- [Running Sync](#running-sync)
|
||||
- [WP-CLI Commands](#wp-cli-commands)
|
||||
- [Cron Setup](#cron-setup)
|
||||
- [Public API](#public-api)
|
||||
- [Database Schema](#database-schema)
|
||||
- [Media Handling](#media-handling)
|
||||
- [Sync Strategy](#sync-strategy)
|
||||
- [Error Recovery](#error-recovery)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
|
||||
## Features
|
||||
|
||||
- Syncs Active and Pending property listings from MLS Grid API
|
||||
- Automatic incremental updates via replication
|
||||
- On-demand image fetching and local caching
|
||||
- Self-healing sync with automatic error recovery
|
||||
- Rate limit compliance (MLS Grid limits enforced)
|
||||
- Resume capability for interrupted syncs
|
||||
- WP-CLI commands for all operations
|
||||
- Public PHP API for theme/plugin integration
|
||||
- Optimized database indexes for search queries
|
||||
|
||||
## Requirements
|
||||
|
||||
- WordPress 5.0+
|
||||
- PHP 7.4+
|
||||
- MySQL 5.7+ or MariaDB 10.2+
|
||||
- WP-CLI (for command-line operations)
|
||||
- MLS Grid API access token
|
||||
|
||||
## Installation
|
||||
|
||||
1. Upload the `mls-by-hansonxyz` folder to `/wp-content/plugins/`
|
||||
2. Activate the plugin through WordPress admin
|
||||
3. Configure API credentials (see Configuration)
|
||||
4. Run initial sync: `wp mls run`
|
||||
|
||||
## Configuration
|
||||
|
||||
### API Credentials
|
||||
|
||||
Add to your `wp-config.php`:
|
||||
|
||||
```php
|
||||
define('MLSGRID_API_URL', 'https://api.mlsgrid.com/v2');
|
||||
define('MLSGRID_ACCESS_TOKEN', 'your-access-token-here');
|
||||
```
|
||||
|
||||
### WordPress Admin Settings
|
||||
|
||||
Navigate to **Settings > MLS Settings** to configure:
|
||||
|
||||
| Setting | Description | Default |
|
||||
|---------|-------------|---------|
|
||||
| Originating System | MLS identifier | `northstar` |
|
||||
| Auto Sync | Enable WP-Cron sync | Disabled |
|
||||
| Sync Interval | WP-Cron frequency | Hourly |
|
||||
|
||||
## Running Sync
|
||||
|
||||
### Smart Sync (Recommended)
|
||||
|
||||
The `wp mls run` command handles all scenarios automatically:
|
||||
|
||||
```bash
|
||||
wp mls run # Smart sync with progress
|
||||
wp mls run --quiet # Status messages only
|
||||
wp mls run --verbose # Full API details
|
||||
wp mls run --silent # For cron (exit code only)
|
||||
```
|
||||
|
||||
**Automatic behavior:**
|
||||
- If no data exists: runs full sync
|
||||
- If data exists: runs incremental sync
|
||||
- If previous sync failed: resumes from checkpoint
|
||||
- If sync already running: safely aborts
|
||||
|
||||
### Manual Sync Commands
|
||||
|
||||
For more control over sync operations:
|
||||
|
||||
```bash
|
||||
# Full sync (Active/Pending properties only)
|
||||
wp mls sync full
|
||||
|
||||
# Incremental sync (changes since last sync)
|
||||
wp mls sync incremental
|
||||
|
||||
# Resume a specific failed sync
|
||||
wp mls sync resume --id=<sync_id>
|
||||
|
||||
# Dry run (no changes)
|
||||
wp mls sync full --dry-run --limit=100
|
||||
```
|
||||
|
||||
### Progress Indicators
|
||||
|
||||
During sync, progress characters indicate activity:
|
||||
|
||||
| Symbol | Meaning |
|
||||
|--------|---------|
|
||||
| `.` | Property created |
|
||||
| `#` | Property updated |
|
||||
| `x` | Property deleted |
|
||||
| `!` | Error occurred |
|
||||
| `\|` | Page complete |
|
||||
|
||||
Use `--verbose` for detailed timestamped output.
|
||||
|
||||
## WP-CLI Commands
|
||||
|
||||
### Testing
|
||||
|
||||
```bash
|
||||
wp mls test connection # Test API connectivity
|
||||
wp mls test auth # Verify authentication
|
||||
```
|
||||
|
||||
### Status and Statistics
|
||||
|
||||
```bash
|
||||
wp mls status # Full status overview
|
||||
wp mls status rate-limits # Rate limit usage only
|
||||
wp mls stats # Database statistics
|
||||
```
|
||||
|
||||
### Sync Operations
|
||||
|
||||
```bash
|
||||
# Smart sync (recommended)
|
||||
wp mls run [--quiet] [--verbose] [--silent]
|
||||
|
||||
# Manual sync
|
||||
wp mls sync full [--dry-run] [--limit=N] [--verbose]
|
||||
wp mls sync incremental [--dry-run] [--verbose]
|
||||
wp mls sync resume --id=<sync_id>
|
||||
```
|
||||
|
||||
### Media Management
|
||||
|
||||
Images are fetched on-demand when properties are viewed. These commands manage the cache:
|
||||
|
||||
```bash
|
||||
wp mls media status # Cache statistics
|
||||
wp mls media fetch --listing=<key> # Pre-cache a listing's images
|
||||
wp mls media fetch --listing=<key> --limit=10
|
||||
wp mls media clear --listing=<key> # Clear cached images
|
||||
```
|
||||
|
||||
### Cache Management
|
||||
|
||||
```bash
|
||||
wp mls cache clear --confirm # Delete ALL synced data
|
||||
wp mls cache cleanup # Remove orphaned media files
|
||||
wp mls cache missing # View failed media downloads
|
||||
wp mls cache missing --clear # Clear the missing media log
|
||||
```
|
||||
|
||||
### Recovery
|
||||
|
||||
```bash
|
||||
wp mls recovery list # Show resumable syncs
|
||||
wp mls recovery auto # Auto-resume most recent failed sync
|
||||
wp mls recovery cleanup # Mark stale syncs as failed
|
||||
```
|
||||
|
||||
## Cron Setup
|
||||
|
||||
### Recommended Setup
|
||||
|
||||
Add to system crontab (`crontab -e`):
|
||||
|
||||
```bash
|
||||
# Smart sync every 15 minutes (handles everything automatically)
|
||||
*/15 * * * * cd /var/www/html && wp mls run --silent --allow-root >> /var/log/mls-sync.log 2>&1
|
||||
```
|
||||
|
||||
This single entry handles:
|
||||
- Initial full sync on first run
|
||||
- Incremental updates on subsequent runs
|
||||
- Automatic recovery from failures
|
||||
- Safe concurrent execution (aborts if already running)
|
||||
|
||||
### Alternative: Manual Control
|
||||
|
||||
```bash
|
||||
# Incremental sync every 15 minutes
|
||||
*/15 * * * * cd /var/www/html && wp mls sync incremental --allow-root >> /var/log/mls-sync.log 2>&1
|
||||
|
||||
# Full rebuild weekly (Sunday 3am)
|
||||
0 3 * * 0 cd /var/www/html && wp mls cache clear --confirm --allow-root && wp mls sync full --allow-root >> /var/log/mls-sync.log 2>&1
|
||||
```
|
||||
|
||||
### Important Notes
|
||||
|
||||
- Use `--allow-root` when running as root
|
||||
- MLS Grid requires refresh at least every 12 hours per IDX rules
|
||||
- Rate limits are handled automatically (plugin waits when approaching limits)
|
||||
- No separate media cron needed - images are fetched on-demand
|
||||
|
||||
## Public API
|
||||
|
||||
### Available Functions
|
||||
|
||||
```php
|
||||
// Get properties with filters
|
||||
$properties = mls_get_properties([
|
||||
'status' => 'Active',
|
||||
'city' => 'Albert Lea',
|
||||
'min_price' => 100000,
|
||||
'max_price' => 500000,
|
||||
'min_beds' => 3,
|
||||
'property_type' => 'Residential',
|
||||
'limit' => 20,
|
||||
'offset' => 0,
|
||||
'orderby' => 'list_price',
|
||||
'order' => 'DESC',
|
||||
]);
|
||||
|
||||
// Get single property by listing key or MLS ID
|
||||
$property = mls_get_property('NST123456');
|
||||
|
||||
// Get primary image (fetches on-demand if not cached)
|
||||
$image_url = mls_get_property_image('NST123456');
|
||||
$image_url = mls_get_property_image('NST123456', false); // Don't fetch, return null if uncached
|
||||
|
||||
// Get all images for a listing
|
||||
$images = mls_get_property_images('NST123456'); // Fetch first 1 if uncached
|
||||
$images = mls_get_property_images('NST123456', 10); // Fetch first 10 if uncached
|
||||
$images = mls_get_property_images('NST123456', 0); // Don't fetch any
|
||||
|
||||
// Get media metadata (no fetching)
|
||||
$media = mls_get_property_media('NST123456');
|
||||
|
||||
// Get distinct cities with listings
|
||||
$cities = mls_get_cities(); // All cities
|
||||
$cities = mls_get_cities('Active'); // Cities with active listings only
|
||||
|
||||
// Get property count
|
||||
$count = mls_get_property_count(['status' => 'Active']);
|
||||
|
||||
// Check if data is available
|
||||
if (mls_is_available()) {
|
||||
// Show property search
|
||||
}
|
||||
|
||||
// Get cache statistics
|
||||
$stats = mls_get_cache_stats();
|
||||
// Returns: ['total_media' => 50000, 'cached' => 1200, 'uncached' => 48800]
|
||||
```
|
||||
|
||||
### Query Parameters
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|-----------|------|-------------|
|
||||
| `status` | string | Active, Pending, Closed |
|
||||
| `property_type` | string | Residential, Land, Commercial, etc. |
|
||||
| `city` | string | City name |
|
||||
| `county` | string | County name |
|
||||
| `postal_code` | string | ZIP code |
|
||||
| `min_price` | int | Minimum list price |
|
||||
| `max_price` | int | Maximum list price |
|
||||
| `min_beds` | int | Minimum bedrooms |
|
||||
| `max_beds` | int | Maximum bedrooms |
|
||||
| `min_baths` | int | Minimum bathrooms |
|
||||
| `min_sqft` | int | Minimum living area |
|
||||
| `max_sqft` | int | Maximum living area |
|
||||
| `year_built_min` | int | Minimum year built |
|
||||
| `year_built_max` | int | Maximum year built |
|
||||
| `listing_key` | string | Specific listing key |
|
||||
| `listing_id` | string | Specific MLS ID |
|
||||
| `search` | string | Search address/remarks |
|
||||
| `limit` | int | Results per page (default: 20) |
|
||||
| `offset` | int | Pagination offset |
|
||||
| `orderby` | string | Sort field |
|
||||
| `order` | string | ASC or DESC |
|
||||
| `include_media` | bool | Include media array |
|
||||
| `fields` | array | Specific fields to return |
|
||||
|
||||
### Property Object Fields
|
||||
|
||||
```php
|
||||
$property->listing_key // Unique identifier
|
||||
$property->listing_id // MLS number
|
||||
$property->standard_status // Active, Pending, Closed
|
||||
$property->list_price // Current price
|
||||
$property->original_list_price
|
||||
$property->close_price
|
||||
|
||||
// Address
|
||||
$property->street_number
|
||||
$property->street_name
|
||||
$property->street_suffix
|
||||
$property->unit_number
|
||||
$property->city
|
||||
$property->state_or_province
|
||||
$property->postal_code
|
||||
$property->county
|
||||
$property->latitude
|
||||
$property->longitude
|
||||
|
||||
// Property details
|
||||
$property->property_type
|
||||
$property->property_sub_type
|
||||
$property->bedrooms_total
|
||||
$property->bathrooms_total
|
||||
$property->bathrooms_full
|
||||
$property->bathrooms_half
|
||||
$property->living_area // Square feet
|
||||
$property->lot_size_area
|
||||
$property->lot_size_units
|
||||
$property->year_built
|
||||
$property->garage_spaces
|
||||
|
||||
// Description
|
||||
$property->public_remarks
|
||||
$property->directions
|
||||
|
||||
// Listing info
|
||||
$property->list_agent_key
|
||||
$property->list_agent_mls_id
|
||||
$property->list_agent_name
|
||||
$property->list_office_key
|
||||
$property->list_office_mls_id
|
||||
$property->list_office_name
|
||||
|
||||
// Dates and timestamps
|
||||
$property->photos_count
|
||||
$property->modification_timestamp
|
||||
$property->photos_change_timestamp
|
||||
$property->listing_contract_date
|
||||
$property->close_date
|
||||
$property->days_on_market
|
||||
$property->created_at
|
||||
$property->updated_at
|
||||
```
|
||||
|
||||
## Database Schema
|
||||
|
||||
### Tables
|
||||
|
||||
All tables use the WordPress prefix (e.g., `wp_mls_properties`).
|
||||
|
||||
#### mls_properties
|
||||
|
||||
Main property listing data. Only Active and Pending properties are stored.
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| id | BIGINT | Auto-increment primary key |
|
||||
| listing_key | VARCHAR(50) | Unique MLS Grid key |
|
||||
| listing_id | VARCHAR(50) | MLS number |
|
||||
| standard_status | VARCHAR(30) | Active, Pending |
|
||||
| list_price | DECIMAL(15,2) | Current price |
|
||||
| city | VARCHAR(100) | City name |
|
||||
| latitude | DECIMAL(10,8) | GPS latitude |
|
||||
| longitude | DECIMAL(11,8) | GPS longitude |
|
||||
| ... | ... | See property fields above |
|
||||
| raw_data | LONGTEXT | Full API response (JSON) |
|
||||
| modification_timestamp | DATETIME | Last modified in MLS |
|
||||
| created_at | DATETIME | Record creation |
|
||||
| updated_at | DATETIME | Record update |
|
||||
|
||||
**Indexes:**
|
||||
- `listing_key` (UNIQUE)
|
||||
- `listing_id`
|
||||
- `standard_status`
|
||||
- `city`
|
||||
- `property_type`
|
||||
- `list_price`
|
||||
- `modification_timestamp`
|
||||
- `bedrooms_total`
|
||||
- `county`
|
||||
- `idx_latitude` - for geo queries
|
||||
- `idx_longitude` - for geo queries
|
||||
- `idx_status_city_price` - composite for search
|
||||
- `idx_status_type` - composite for filtering
|
||||
|
||||
#### mls_media
|
||||
|
||||
Media metadata and cache status. Images are downloaded on-demand.
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| id | BIGINT | Auto-increment primary key |
|
||||
| listing_key | VARCHAR(50) | Property reference |
|
||||
| media_key | VARCHAR(100) | Unique media identifier |
|
||||
| media_type | VARCHAR(30) | Photo, Document, etc. |
|
||||
| media_order | INT | Display order |
|
||||
| media_url | VARCHAR(1000) | Original MLS Grid URL |
|
||||
| local_path | VARCHAR(500) | Cached file path |
|
||||
| local_url | VARCHAR(500) | Cached file URL |
|
||||
| downloaded_at | DATETIME | When cached |
|
||||
|
||||
#### mls_sync_state
|
||||
|
||||
Sync progress tracking for resume capability.
|
||||
|
||||
| Column | Type | Description |
|
||||
|--------|------|-------------|
|
||||
| id | BIGINT | Sync operation ID |
|
||||
| sync_type | VARCHAR(30) | full, incremental |
|
||||
| status | VARCHAR(20) | pending, running, completed, failed |
|
||||
| last_next_link | VARCHAR(2000) | Resume checkpoint |
|
||||
| records_processed | INT | Total processed |
|
||||
| records_created | INT | New records |
|
||||
| records_updated | INT | Updated records |
|
||||
| records_deleted | INT | Deleted records |
|
||||
|
||||
#### mls_rate_limits
|
||||
|
||||
API rate limit tracking.
|
||||
|
||||
#### mls_sync_log
|
||||
|
||||
Debug logging for sync operations.
|
||||
|
||||
#### mls_media_log
|
||||
|
||||
Media download audit trail.
|
||||
|
||||
## Media Handling
|
||||
|
||||
### On-Demand Fetching
|
||||
|
||||
Per MLS Grid rules, media URLs cannot be used directly on websites. Images must be downloaded and served from your own server.
|
||||
|
||||
**How it works:**
|
||||
|
||||
1. Property sync stores media metadata (URLs, keys, order) but does NOT download images
|
||||
2. When `mls_get_property_image()` is called, the image is fetched and cached locally
|
||||
3. Subsequent requests serve from local cache
|
||||
4. Cache location: `wp-content/uploads/mls-listings/{prefix}/{listing_key}/`
|
||||
|
||||
**Benefits:**
|
||||
- No rate limit issues from bulk downloading
|
||||
- Images cached only when needed
|
||||
- Automatic re-fetch if cache cleared
|
||||
- Works with MLS Grid's URL expiration
|
||||
|
||||
### Pre-caching Images
|
||||
|
||||
To pre-cache images for specific listings:
|
||||
|
||||
```bash
|
||||
wp mls media fetch --listing=NST123456 --limit=10
|
||||
```
|
||||
|
||||
### Cache Statistics
|
||||
|
||||
```bash
|
||||
wp mls media status
|
||||
```
|
||||
|
||||
Shows total media records, cached count, and uncached count.
|
||||
|
||||
## Sync Strategy
|
||||
|
||||
### Initial Import (Full Sync)
|
||||
|
||||
- Fetches ONLY Active and Pending properties
|
||||
- Filter: `MlgCanView eq true AND (StandardStatus eq 'Active' OR StandardStatus eq 'Pending')`
|
||||
- Uses `@odata.nextLink` for pagination (NOT `$skip`)
|
||||
- Approximately 30,000 records for NorthStar MLS
|
||||
- Takes 30-45 minutes on first run
|
||||
|
||||
### Replication (Incremental Sync)
|
||||
|
||||
- Fetches ALL properties modified since last sync
|
||||
- No filter on status (need to detect changes)
|
||||
- For each record:
|
||||
- If `MlgCanView = false`: DELETE from local DB
|
||||
- If `StandardStatus` not Active/Pending: DELETE from local DB
|
||||
- Otherwise: INSERT or UPDATE
|
||||
|
||||
### Why This Approach?
|
||||
|
||||
1. MLS Grid API limits `$skip` to ~80,000 - bulk scanning fails
|
||||
2. Only Active/Pending properties needed for display
|
||||
3. Replication is efficient - only fetches changes
|
||||
4. Proper deletion handling when properties sell
|
||||
|
||||
## Error Recovery
|
||||
|
||||
### Automatic Recovery
|
||||
|
||||
The plugin saves progress after each API page. If a sync fails:
|
||||
|
||||
1. Progress is preserved in `mls_sync_state` table
|
||||
2. Next `wp mls run` automatically resumes from checkpoint
|
||||
3. Failed syncs older than 1 hour are marked for resume
|
||||
|
||||
### Manual Recovery
|
||||
|
||||
```bash
|
||||
# View resumable syncs
|
||||
wp mls recovery list
|
||||
|
||||
# Auto-resume most recent
|
||||
wp mls recovery auto
|
||||
|
||||
# Resume specific sync
|
||||
wp mls sync resume --id=<sync_id>
|
||||
|
||||
# Mark stale syncs as failed
|
||||
wp mls recovery cleanup
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Connection Failed
|
||||
|
||||
```bash
|
||||
wp mls test connection
|
||||
wp mls test auth
|
||||
```
|
||||
|
||||
Check:
|
||||
- API token in wp-config.php
|
||||
- Network connectivity
|
||||
- MLS Grid API status
|
||||
|
||||
### No Data After Sync
|
||||
|
||||
```bash
|
||||
wp mls status
|
||||
wp mls stats
|
||||
```
|
||||
|
||||
Check:
|
||||
- Rate limits (may need to wait)
|
||||
- WordPress debug log for API errors
|
||||
- Sync state for failures
|
||||
|
||||
### Media Not Loading
|
||||
|
||||
```bash
|
||||
wp mls media status
|
||||
```
|
||||
|
||||
Check:
|
||||
- Upload directory permissions
|
||||
- Disk space
|
||||
- MLS Grid media URL validity
|
||||
|
||||
### Sync Taking Too Long
|
||||
|
||||
Initial sync of ~30K properties takes 30-45 minutes. Use `--verbose` to monitor progress.
|
||||
|
||||
### Rate Limit Exceeded
|
||||
|
||||
The plugin automatically waits when approaching limits. If persistent:
|
||||
- Reduce sync frequency
|
||||
- Check for other API consumers
|
||||
- Contact MLS Grid support
|
||||
|
||||
### Clearing Data
|
||||
|
||||
To start fresh:
|
||||
|
||||
```bash
|
||||
wp mls cache clear --confirm
|
||||
wp mls run
|
||||
```
|
||||
|
||||
### Database Issues
|
||||
|
||||
If indexes are missing, trigger recreation:
|
||||
|
||||
```bash
|
||||
wp eval "MLS_DB::create_tables();"
|
||||
```
|
||||
|
||||
## Rate Limits
|
||||
|
||||
MLS Grid enforces these limits:
|
||||
|
||||
| Limit | Value |
|
||||
|-------|-------|
|
||||
| Per second | 2 requests |
|
||||
| Per hour | 7,200 requests |
|
||||
| Per day | 40,000 requests |
|
||||
| Data per hour | 4 GB |
|
||||
|
||||
The plugin automatically:
|
||||
- Waits 500ms between requests
|
||||
- Tracks hourly/daily usage
|
||||
- Pauses when approaching limits
|
||||
- Retries with exponential backoff on 429 errors
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
mls-by-hansonxyz/
|
||||
├── mls-by-hansonxyz.php # Main plugin file, public API
|
||||
├── uninstall.php # Cleanup on uninstall
|
||||
├── README.md # This file
|
||||
├── admin/
|
||||
│ └── class-mls-admin.php # WordPress admin interface
|
||||
├── cli/
|
||||
│ └── class-mls-cli.php # WP-CLI commands
|
||||
├── includes/
|
||||
│ ├── class-mls-activator.php # Plugin activation
|
||||
│ ├── class-mls-api-client.php # MLS Grid API communication
|
||||
│ ├── class-mls-db.php # Database operations
|
||||
│ ├── class-mls-deactivator.php # Plugin deactivation
|
||||
│ ├── class-mls-logger.php # Event logging
|
||||
│ ├── class-mls-media-handler.php # On-demand image caching
|
||||
│ ├── class-mls-options.php # Configuration management
|
||||
│ ├── class-mls-query.php # Public query API
|
||||
│ ├── class-mls-rate-limiter.php # Rate limit compliance
|
||||
│ └── class-mls-sync-engine.php # Sync orchestration
|
||||
└── docs/
|
||||
├── API.md # MLS Grid API reference
|
||||
├── CLAUDE.md # AI assistant context
|
||||
└── USAGE.md # User documentation
|
||||
```
|
||||
|
||||
## Support
|
||||
|
||||
- Plugin logs: Settings > MLS Settings in WordPress admin
|
||||
- Debug log: `wp-content/debug.log` (if WP_DEBUG enabled)
|
||||
- MLS Grid API: support@mlsgrid.com
|
||||
|
||||
## License
|
||||
|
||||
GPL-2.0+
|
||||
@@ -34,6 +34,7 @@ class MLS_CLI {
|
||||
WP_CLI::add_command('mls test', array($instance, 'test'));
|
||||
WP_CLI::add_command('mls status', array($instance, 'status'));
|
||||
WP_CLI::add_command('mls sync', array($instance, 'sync'));
|
||||
WP_CLI::add_command('mls run', array($instance, 'run'));
|
||||
WP_CLI::add_command('mls stats', array($instance, 'stats'));
|
||||
WP_CLI::add_command('mls cache', array($instance, 'cache'));
|
||||
WP_CLI::add_command('mls recovery', array($instance, 'recovery'));
|
||||
@@ -369,6 +370,149 @@ class MLS_CLI {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Run smart sync - autonomous self-healing sync.
|
||||
*
|
||||
* This is the recommended command for automated/cron usage. It automatically
|
||||
* determines the best action based on current state:
|
||||
*
|
||||
* - If a sync is running: abort (prevents duplicate syncs)
|
||||
* - If a previous sync failed/interrupted: resume it
|
||||
* - If no data exists: run full sync
|
||||
* - Otherwise: run incremental sync
|
||||
*
|
||||
* Failed syncs are automatically recoverable on the next run.
|
||||
*
|
||||
* ## OPTIONS
|
||||
*
|
||||
* [--quiet]
|
||||
* : Suppress progress output (still shows status messages)
|
||||
*
|
||||
* [--verbose]
|
||||
* : Show detailed output including API requests
|
||||
*
|
||||
* [--silent]
|
||||
* : Suppress all output except errors (for cron)
|
||||
*
|
||||
* ## EXAMPLES
|
||||
*
|
||||
* wp mls run # Smart sync with progress
|
||||
* wp mls run --quiet # Smart sync, status only
|
||||
* wp mls run --verbose # Smart sync with full details
|
||||
* wp mls run --silent # For cron jobs
|
||||
*
|
||||
* @subcommand run
|
||||
*/
|
||||
public function run($args, $assoc_args) {
|
||||
$quiet = isset($assoc_args['quiet']);
|
||||
$verbose = isset($assoc_args['verbose']);
|
||||
$silent = isset($assoc_args['silent']);
|
||||
|
||||
$sync_engine = $this->plugin->get_sync_engine();
|
||||
|
||||
// Status callback for high-level messages
|
||||
$status_callback = null;
|
||||
if (!$silent) {
|
||||
$status_callback = function($message, $level = 'info') {
|
||||
$timestamp = date('H:i:s');
|
||||
switch ($level) {
|
||||
case 'warning':
|
||||
WP_CLI::warning("[{$timestamp}] {$message}");
|
||||
break;
|
||||
case 'error':
|
||||
WP_CLI::warning("[{$timestamp}] {$message}");
|
||||
break;
|
||||
default:
|
||||
WP_CLI::line("[{$timestamp}] {$message}");
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
// Progress callback for record-level progress
|
||||
$progress_callback = null;
|
||||
if (!$quiet && !$silent) {
|
||||
$progress_callback = function($event, $data = array()) use ($verbose) {
|
||||
if ($verbose) {
|
||||
$this->output_verbose_event($event, $data);
|
||||
} else {
|
||||
switch ($event) {
|
||||
case 'property_created':
|
||||
echo '.';
|
||||
break;
|
||||
case 'property_updated':
|
||||
echo '#';
|
||||
break;
|
||||
case 'property_deleted':
|
||||
echo 'x';
|
||||
break;
|
||||
case 'property_error':
|
||||
echo '!';
|
||||
break;
|
||||
case 'page_complete':
|
||||
echo '|';
|
||||
break;
|
||||
}
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
if (!$silent) {
|
||||
WP_CLI::line('');
|
||||
WP_CLI::line('=== MLS Smart Sync ===');
|
||||
WP_CLI::line('');
|
||||
}
|
||||
|
||||
// Run smart sync
|
||||
$result = $sync_engine->smart_sync($progress_callback, $status_callback);
|
||||
|
||||
// Handle aborted case (sync already running)
|
||||
if (isset($result['action']) && $result['action'] === 'aborted') {
|
||||
if (!$silent) {
|
||||
WP_CLI::warning('Sync aborted: ' . ($result['reason'] ?? 'Unknown reason'));
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
// Output newline after progress dots
|
||||
if (!$quiet && !$silent && !$verbose) {
|
||||
echo "\n";
|
||||
}
|
||||
|
||||
// Output results
|
||||
if (!$silent) {
|
||||
$action_labels = array(
|
||||
'full' => 'Full sync',
|
||||
'incremental' => 'Incremental sync',
|
||||
'resumed' => 'Resumed sync',
|
||||
);
|
||||
$action_label = $action_labels[$result['action']] ?? 'Sync';
|
||||
|
||||
if ($result['success']) {
|
||||
WP_CLI::success("{$action_label} completed successfully!");
|
||||
} else {
|
||||
WP_CLI::warning("{$action_label} failed: " . ($result['error'] ?? 'Unknown error'));
|
||||
WP_CLI::line('The sync can be resumed on the next run.');
|
||||
}
|
||||
|
||||
if (isset($result['stats'])) {
|
||||
$stats = $result['stats'];
|
||||
WP_CLI::line(sprintf(
|
||||
'Processed: %d | Created: %d | Updated: %d | Deleted: %d | Errors: %d',
|
||||
$stats['processed'],
|
||||
$stats['created'],
|
||||
$stats['updated'],
|
||||
$stats['deleted'],
|
||||
$stats['errors']
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
// Exit with error code if failed (for cron monitoring)
|
||||
if (!$result['success']) {
|
||||
WP_CLI::halt(1);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Print progress legend
|
||||
*
|
||||
|
||||
@@ -63,7 +63,13 @@ wp mls test auth
|
||||
wp mls status
|
||||
wp mls status rate-limits
|
||||
|
||||
# Run property sync
|
||||
# SMART SYNC (recommended for automation)
|
||||
wp mls run # Auto-detect: full, incremental, or resume
|
||||
wp mls run --quiet # Status messages only
|
||||
wp mls run --silent # For cron (exit code only)
|
||||
wp mls run --verbose # Full details
|
||||
|
||||
# Manual sync commands (for more control)
|
||||
wp mls sync full [--dry-run] [--limit=N] [--verbose] # Initial: Active/Pending only
|
||||
wp mls sync incremental [--dry-run] [--verbose] # Replication: all changes
|
||||
wp mls sync resume --id=<sync_id>
|
||||
@@ -171,7 +177,20 @@ The sync engine saves progress after each page:
|
||||
### Recommended Cron Setup
|
||||
|
||||
```bash
|
||||
# Replication sync every 15 minutes (MLS Grid recommended)
|
||||
# Smart sync every 15 minutes (recommended - handles everything automatically)
|
||||
*/15 * * * * cd /var/www/html && wp mls run --silent --allow-root >> /var/log/mls-sync.log 2>&1
|
||||
```
|
||||
|
||||
The `wp mls run` command automatically:
|
||||
- Runs full sync if no data exists
|
||||
- Runs incremental sync if data exists
|
||||
- Resumes failed/interrupted syncs
|
||||
- Aborts safely if another sync is running
|
||||
|
||||
For manual control, use individual commands:
|
||||
|
||||
```bash
|
||||
# Replication sync every 15 minutes
|
||||
*/15 * * * * cd /var/www/html && wp mls sync incremental --allow-root >> /var/log/mls-sync.log 2>&1
|
||||
|
||||
# Full re-sync weekly (Sunday 3am) - rebuilds from scratch
|
||||
|
||||
@@ -43,6 +43,29 @@ Navigate to **Settings > MLS Settings** to configure:
|
||||
|
||||
### Via WP-CLI
|
||||
|
||||
#### Smart Sync (Recommended)
|
||||
|
||||
The `wp mls run` command is the recommended way to sync. It automatically handles all scenarios:
|
||||
|
||||
```bash
|
||||
wp mls run # Smart sync with progress
|
||||
wp mls run --quiet # Status messages only
|
||||
wp mls run --verbose # Full API details
|
||||
wp mls run --silent # For cron (no output)
|
||||
```
|
||||
|
||||
**What it does automatically:**
|
||||
- If a sync is already running: aborts (prevents duplicates)
|
||||
- If a previous sync failed/interrupted: resumes it
|
||||
- If no data exists: runs full sync
|
||||
- Otherwise: runs incremental sync
|
||||
|
||||
Failed syncs are automatically resumed on the next run - no manual intervention needed.
|
||||
|
||||
#### Manual Sync Commands
|
||||
|
||||
For more control, use the individual sync commands:
|
||||
|
||||
```bash
|
||||
# Test connection first
|
||||
wp mls test connection
|
||||
@@ -54,9 +77,6 @@ wp mls sync full
|
||||
# Run incremental updates
|
||||
wp mls sync incremental
|
||||
|
||||
# Download pending media
|
||||
wp mls sync media
|
||||
|
||||
# Use --verbose for detailed output
|
||||
wp mls sync full --verbose
|
||||
wp mls sync incremental --verbose
|
||||
@@ -80,17 +100,25 @@ Use `--verbose` for detailed timestamped output showing API requests and individ
|
||||
Add to your system crontab (`crontab -e`) for scheduled sync:
|
||||
|
||||
```bash
|
||||
# Incremental sync every hour (recommended for production)
|
||||
# Smart sync every 15 minutes (recommended)
|
||||
# Automatically handles: initial sync, incremental updates, and error recovery
|
||||
*/15 * * * * cd /var/www/html && wp mls run --silent --allow-root >> /var/log/mls-sync.log 2>&1
|
||||
```
|
||||
|
||||
That's it! The `wp mls run` command handles everything automatically:
|
||||
- First run: performs full initial sync
|
||||
- Subsequent runs: performs incremental sync
|
||||
- After failures: resumes from where it left off
|
||||
- Concurrent runs: safely aborts if another sync is running
|
||||
|
||||
**For more control**, use the individual commands:
|
||||
|
||||
```bash
|
||||
# Incremental sync every hour
|
||||
0 * * * * cd /var/www/html && wp mls sync incremental --allow-root >> /var/log/mls-sync.log 2>&1
|
||||
|
||||
# Or every 30 minutes for more frequent updates
|
||||
*/30 * * * * cd /var/www/html && wp mls sync incremental --allow-root >> /var/log/mls-sync.log 2>&1
|
||||
|
||||
# Full sync weekly (Sunday at 3am) to catch any missed records
|
||||
# Full sync weekly (Sunday at 3am) to rebuild from scratch
|
||||
0 3 * * 0 cd /var/www/html && wp mls sync full --allow-root >> /var/log/mls-sync.log 2>&1
|
||||
|
||||
# Download any pending media every 15 minutes
|
||||
*/15 * * * * cd /var/www/html && wp mls sync media --limit=50 --allow-root >> /var/log/mls-sync.log 2>&1
|
||||
```
|
||||
|
||||
**Important Notes:**
|
||||
@@ -98,6 +126,7 @@ Add to your system crontab (`crontab -e`) for scheduled sync:
|
||||
- Redirect output to a log file for debugging
|
||||
- MLS Grid requires refresh at least every 12 hours per IDX rules
|
||||
- The plugin handles rate limits automatically (waits if approaching limits)
|
||||
- Media images are fetched on-demand when properties are viewed (no separate cron needed)
|
||||
|
||||
### Via WP-Cron (Alternative)
|
||||
|
||||
|
||||
@@ -9,6 +9,12 @@ if (!defined('ABSPATH')) {
|
||||
|
||||
class MLS_DB {
|
||||
|
||||
/**
|
||||
* Schema version for index migrations
|
||||
* Increment this when adding new indexes
|
||||
*/
|
||||
const SCHEMA_VERSION = 2;
|
||||
|
||||
/**
|
||||
* Get table name with prefix
|
||||
*
|
||||
@@ -275,6 +281,78 @@ class MLS_DB {
|
||||
) {$charset_collate};";
|
||||
|
||||
dbDelta($sql_media_log);
|
||||
|
||||
// Run index migrations
|
||||
self::run_index_migrations();
|
||||
}
|
||||
|
||||
/**
|
||||
* Run index migrations that dbDelta cannot handle
|
||||
*
|
||||
* dbDelta can create tables and add columns, but cannot add indexes
|
||||
* to existing tables. This method handles incremental index additions.
|
||||
*/
|
||||
public static function run_index_migrations() {
|
||||
global $wpdb;
|
||||
|
||||
$current_schema = (int) get_option('mls_schema_version', 1);
|
||||
|
||||
// Migration to schema version 2: Add search and geo indexes
|
||||
if ($current_schema < 2) {
|
||||
$table_properties = $wpdb->prefix . MLS_TABLE_PROPERTIES;
|
||||
$table_media = $wpdb->prefix . MLS_TABLE_MEDIA;
|
||||
|
||||
// Check and add indexes only if they don't exist
|
||||
$existing_indexes = self::get_existing_indexes($table_properties);
|
||||
|
||||
// Geospatial indexes for bounding box queries
|
||||
if (!isset($existing_indexes['idx_latitude'])) {
|
||||
$wpdb->query("ALTER TABLE {$table_properties} ADD INDEX idx_latitude (latitude)");
|
||||
}
|
||||
if (!isset($existing_indexes['idx_longitude'])) {
|
||||
$wpdb->query("ALTER TABLE {$table_properties} ADD INDEX idx_longitude (longitude)");
|
||||
}
|
||||
|
||||
// Composite index for common search pattern (status + city + price)
|
||||
if (!isset($existing_indexes['idx_status_city_price'])) {
|
||||
$wpdb->query("ALTER TABLE {$table_properties} ADD INDEX idx_status_city_price (standard_status, city, list_price)");
|
||||
}
|
||||
|
||||
// Composite index for status + property_type searches
|
||||
if (!isset($existing_indexes['idx_status_type'])) {
|
||||
$wpdb->query("ALTER TABLE {$table_properties} ADD INDEX idx_status_type (standard_status, property_type)");
|
||||
}
|
||||
|
||||
// Media table: composite index for listing + order (eliminates filesort)
|
||||
$media_indexes = self::get_existing_indexes($table_media);
|
||||
if (!isset($media_indexes['idx_listing_order'])) {
|
||||
$wpdb->query("ALTER TABLE {$table_media} ADD INDEX idx_listing_order (listing_key, media_order)");
|
||||
}
|
||||
|
||||
update_option('mls_schema_version', 2);
|
||||
}
|
||||
|
||||
// Future migrations go here:
|
||||
// if ($current_schema < 3) { ... }
|
||||
}
|
||||
|
||||
/**
|
||||
* Get existing indexes for a table
|
||||
*
|
||||
* @param string $table Full table name
|
||||
* @return array Index names as keys
|
||||
*/
|
||||
private static function get_existing_indexes($table) {
|
||||
global $wpdb;
|
||||
|
||||
$indexes = array();
|
||||
$results = $wpdb->get_results("SHOW INDEX FROM {$table}");
|
||||
|
||||
foreach ($results as $row) {
|
||||
$indexes[$row->Key_name] = true;
|
||||
}
|
||||
|
||||
return $indexes;
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -937,4 +937,113 @@ class MLS_Sync_Engine {
|
||||
|
||||
return $this->resume_sync($resumable->id, $progress_callback);
|
||||
}
|
||||
|
||||
/**
|
||||
* Smart sync - autonomous self-healing sync that handles all scenarios
|
||||
*
|
||||
* Decision logic:
|
||||
* 1. If a sync is currently running (and not stale), abort
|
||||
* 2. If there's a resumable failed/interrupted sync, resume it
|
||||
* 3. If no data exists, run full sync
|
||||
* 4. Otherwise, run incremental sync
|
||||
*
|
||||
* On failure, the sync state is preserved for future resume.
|
||||
*
|
||||
* @param callable|null $progress_callback Progress callback
|
||||
* @param callable|null $status_callback Callback for status messages: function(string $message, string $level)
|
||||
* @return array Sync results with 'action' key indicating what was done
|
||||
*/
|
||||
public function smart_sync($progress_callback = null, $status_callback = null) {
|
||||
// Helper to emit status messages
|
||||
$status = function($message, $level = 'info') use ($status_callback) {
|
||||
if ($status_callback) {
|
||||
call_user_func($status_callback, $message, $level);
|
||||
}
|
||||
$this->logger->log($level, $message);
|
||||
};
|
||||
|
||||
// Step 1: Clean up stale syncs (running > 1 hour = probably dead)
|
||||
$stale_cleaned = $this->cleanup_stale_syncs();
|
||||
if ($stale_cleaned > 0) {
|
||||
$status("Cleaned up {$stale_cleaned} stale sync(s)", 'info');
|
||||
}
|
||||
|
||||
// Step 2: Check if a sync is actively running
|
||||
$running = $this->get_running_sync();
|
||||
if ($running) {
|
||||
$status("Sync #{$running->id} is already running (started {$running->started_at})", 'warning');
|
||||
return array(
|
||||
'success' => false,
|
||||
'action' => 'aborted',
|
||||
'reason' => 'Sync already running',
|
||||
'running_sync' => $running,
|
||||
);
|
||||
}
|
||||
|
||||
// Step 3: Check for resumable syncs
|
||||
$resumable = $this->get_latest_resumable();
|
||||
if ($resumable) {
|
||||
$status("Found resumable sync #{$resumable->id} ({$resumable->sync_type}), processed {$resumable->records_processed} records", 'info');
|
||||
$status("Resuming...", 'info');
|
||||
|
||||
$result = $this->resume_sync($resumable->id, $progress_callback);
|
||||
$result['action'] = 'resumed';
|
||||
$result['resumed_sync_id'] = $resumable->id;
|
||||
return $result;
|
||||
}
|
||||
|
||||
// Step 4: Check if we have any data
|
||||
$has_data = $this->has_synced_data();
|
||||
|
||||
if (!$has_data) {
|
||||
// No data - need full sync
|
||||
$status("No existing data found, starting full sync", 'info');
|
||||
|
||||
$result = $this->run_full_sync(false, null, $progress_callback);
|
||||
$result['action'] = 'full';
|
||||
return $result;
|
||||
}
|
||||
|
||||
// Step 5: We have data - run incremental sync
|
||||
$last_timestamp = $this->get_last_modification_timestamp();
|
||||
$status("Running incremental sync (changes since {$last_timestamp})", 'info');
|
||||
|
||||
$result = $this->run_incremental_sync(false, $progress_callback);
|
||||
$result['action'] = 'incremental';
|
||||
return $result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if there's a currently running sync (not stale)
|
||||
*
|
||||
* @return object|null Running sync state or null
|
||||
*/
|
||||
public function get_running_sync() {
|
||||
global $wpdb;
|
||||
|
||||
$one_hour_ago = date('Y-m-d H:i:s', strtotime('-1 hour'));
|
||||
|
||||
return $wpdb->get_row($wpdb->prepare(
|
||||
"SELECT * FROM {$this->db->sync_state_table()}
|
||||
WHERE status = 'running' AND updated_at >= %s
|
||||
ORDER BY started_at DESC
|
||||
LIMIT 1",
|
||||
$one_hour_ago
|
||||
));
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if we have any synced property data
|
||||
*
|
||||
* @return bool
|
||||
*/
|
||||
public function has_synced_data() {
|
||||
global $wpdb;
|
||||
|
||||
$count = $wpdb->get_var(
|
||||
"SELECT COUNT(*) FROM {$this->db->properties_table()} WHERE mlg_can_view = 1"
|
||||
);
|
||||
|
||||
return (int) $count > 0;
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user