Add WebP conversion and garbage collection to MLS plugin
Image handling improvements: - Convert PNG and images >500KB to WebP format - Resize images wider than 1600px maintaining aspect ratio - Check for .webp version before falling back to original - WebP quality set to 80 (equivalent to JPEG 90%) Garbage collection for disk space management: - New MLS_Garbage_Collector class runs after each sync - Only active when MLS_GC_DISK_THRESHOLD defined in wp-config - Deletes image directories older than 24 hours, oldest first - Stops when free space reaches 5GB or 2GB deleted per run - Protects recently accessed images from deletion Documentation: - Added Garbage Collection section to README - Updated Features list and File Structure Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -14,6 +14,7 @@ WordPress plugin for syncing MLS Grid API data (NorthStar MLS) into a local data
|
|||||||
- [Public API](#public-api)
|
- [Public API](#public-api)
|
||||||
- [Database Schema](#database-schema)
|
- [Database Schema](#database-schema)
|
||||||
- [Media Handling](#media-handling)
|
- [Media Handling](#media-handling)
|
||||||
|
- [Garbage Collection](#garbage-collection)
|
||||||
- [Sync Strategy](#sync-strategy)
|
- [Sync Strategy](#sync-strategy)
|
||||||
- [Error Recovery](#error-recovery)
|
- [Error Recovery](#error-recovery)
|
||||||
- [Troubleshooting](#troubleshooting)
|
- [Troubleshooting](#troubleshooting)
|
||||||
@@ -23,6 +24,8 @@ WordPress plugin for syncing MLS Grid API data (NorthStar MLS) into a local data
|
|||||||
- Syncs Active and Pending property listings from MLS Grid API
|
- Syncs Active and Pending property listings from MLS Grid API
|
||||||
- Automatic incremental updates via replication
|
- Automatic incremental updates via replication
|
||||||
- On-demand image fetching and local caching
|
- On-demand image fetching and local caching
|
||||||
|
- Automatic WebP conversion for cached images
|
||||||
|
- Disk space garbage collection for image cache
|
||||||
- Self-healing sync with automatic error recovery
|
- Self-healing sync with automatic error recovery
|
||||||
- Rate limit compliance (MLS Grid limits enforced)
|
- Rate limit compliance (MLS Grid limits enforced)
|
||||||
- Resume capability for interrupted syncs
|
- Resume capability for interrupted syncs
|
||||||
@@ -56,6 +59,17 @@ define('MLSGRID_API_URL', 'https://api.mlsgrid.com/v2');
|
|||||||
define('MLSGRID_ACCESS_TOKEN', 'your-access-token-here');
|
define('MLSGRID_ACCESS_TOKEN', 'your-access-token-here');
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Image Garbage Collection (Optional)
|
||||||
|
|
||||||
|
To enable automatic cleanup of old cached images when disk space is low, add to `wp-config.php`:
|
||||||
|
|
||||||
|
```php
|
||||||
|
// Enable garbage collection when free space drops below 5GB
|
||||||
|
define('MLS_GC_DISK_THRESHOLD', 5 * 1024 * 1024 * 1024); // 5GB in bytes
|
||||||
|
```
|
||||||
|
|
||||||
|
See [Garbage Collection](#garbage-collection) for details.
|
||||||
|
|
||||||
### WordPress Admin Settings
|
### WordPress Admin Settings
|
||||||
|
|
||||||
Navigate to **Settings > MLS Settings** to configure:
|
Navigate to **Settings > MLS Settings** to configure:
|
||||||
@@ -464,6 +478,80 @@ wp mls media status
|
|||||||
|
|
||||||
Shows total media records, cached count, and uncached count.
|
Shows total media records, cached count, and uncached count.
|
||||||
|
|
||||||
|
## Garbage Collection
|
||||||
|
|
||||||
|
The plugin includes automatic garbage collection to prevent disk space from filling up with cached MLS images.
|
||||||
|
|
||||||
|
### Enabling Garbage Collection
|
||||||
|
|
||||||
|
Add to `wp-config.php`:
|
||||||
|
|
||||||
|
```php
|
||||||
|
// Enable garbage collection when free space drops below 5GB
|
||||||
|
define('MLS_GC_DISK_THRESHOLD', 5 * 1024 * 1024 * 1024); // 5GB in bytes
|
||||||
|
```
|
||||||
|
|
||||||
|
If `MLS_GC_DISK_THRESHOLD` is not defined, garbage collection is disabled.
|
||||||
|
|
||||||
|
### How It Works
|
||||||
|
|
||||||
|
1. After each sync (`wp mls run`), the plugin checks free disk space on the volume hosting MLS images
|
||||||
|
2. If free space is below the threshold, cleanup begins
|
||||||
|
3. Directories older than 24 hours are deleted, oldest first
|
||||||
|
4. Cleanup stops when:
|
||||||
|
- Free space reaches 5GB, OR
|
||||||
|
- 2GB has been deleted in this run
|
||||||
|
5. Directories modified within the last 24 hours are never deleted (protects recently accessed images)
|
||||||
|
|
||||||
|
### Behavior Summary
|
||||||
|
|
||||||
|
| Setting | Value |
|
||||||
|
|---------|-------|
|
||||||
|
| Threshold trigger | Configurable via `MLS_GC_DISK_THRESHOLD` |
|
||||||
|
| Target free space | 5GB |
|
||||||
|
| Max delete per run | 2GB |
|
||||||
|
| Minimum directory age | 24 hours |
|
||||||
|
| Runs automatically | After every sync |
|
||||||
|
|
||||||
|
### CLI Output
|
||||||
|
|
||||||
|
During sync, garbage collection status is shown:
|
||||||
|
|
||||||
|
```
|
||||||
|
Garbage Collection:
|
||||||
|
Disk space OK: 12.45 GB free (threshold: 5.00 GB)
|
||||||
|
```
|
||||||
|
|
||||||
|
Or if cleanup occurs:
|
||||||
|
|
||||||
|
```
|
||||||
|
Garbage Collection:
|
||||||
|
Disk space low: 3.21 GB free (threshold: 5.00 GB). Starting cleanup...
|
||||||
|
Deleted: NST123456 (45.23 MB)
|
||||||
|
Deleted: NST789012 (38.91 MB)
|
||||||
|
...
|
||||||
|
Cleanup complete: Deleted 42 directories (1.89 GB). Free space now: 5.10 GB
|
||||||
|
```
|
||||||
|
|
||||||
|
### Recommended Threshold
|
||||||
|
|
||||||
|
For most installations, 5GB is a good threshold:
|
||||||
|
|
||||||
|
```php
|
||||||
|
define('MLS_GC_DISK_THRESHOLD', 5 * 1024 * 1024 * 1024);
|
||||||
|
```
|
||||||
|
|
||||||
|
For servers with limited disk space, you may want a higher threshold to trigger cleanup earlier:
|
||||||
|
|
||||||
|
```php
|
||||||
|
// Trigger cleanup when below 10GB
|
||||||
|
define('MLS_GC_DISK_THRESHOLD', 10 * 1024 * 1024 * 1024);
|
||||||
|
```
|
||||||
|
|
||||||
|
### Image Regeneration
|
||||||
|
|
||||||
|
When a deleted image is requested again, it is automatically re-fetched from MLS Grid and cached. This is the normal on-demand fetching behavior - garbage collection simply clears old cached files to free disk space.
|
||||||
|
|
||||||
## Sync Strategy
|
## Sync Strategy
|
||||||
|
|
||||||
### Initial Import (Full Sync)
|
### Initial Import (Full Sync)
|
||||||
@@ -610,16 +698,17 @@ mls-by-hansonxyz/
|
|||||||
├── cli/
|
├── cli/
|
||||||
│ └── class-mls-cli.php # WP-CLI commands
|
│ └── class-mls-cli.php # WP-CLI commands
|
||||||
├── includes/
|
├── includes/
|
||||||
│ ├── class-mls-activator.php # Plugin activation
|
│ ├── class-mls-activator.php # Plugin activation
|
||||||
│ ├── class-mls-api-client.php # MLS Grid API communication
|
│ ├── class-mls-api-client.php # MLS Grid API communication
|
||||||
│ ├── class-mls-db.php # Database operations
|
│ ├── class-mls-db.php # Database operations
|
||||||
│ ├── class-mls-deactivator.php # Plugin deactivation
|
│ ├── class-mls-deactivator.php # Plugin deactivation
|
||||||
│ ├── class-mls-logger.php # Event logging
|
│ ├── class-mls-garbage-collector.php # Disk space management
|
||||||
│ ├── class-mls-media-handler.php # On-demand image caching
|
│ ├── class-mls-logger.php # Event logging
|
||||||
│ ├── class-mls-options.php # Configuration management
|
│ ├── class-mls-media-handler.php # On-demand image caching
|
||||||
│ ├── class-mls-query.php # Public query API
|
│ ├── class-mls-options.php # Configuration management
|
||||||
│ ├── class-mls-rate-limiter.php # Rate limit compliance
|
│ ├── class-mls-query.php # Public query API
|
||||||
│ └── class-mls-sync-engine.php # Sync orchestration
|
│ ├── class-mls-rate-limiter.php # Rate limit compliance
|
||||||
|
│ └── class-mls-sync-engine.php # Sync orchestration
|
||||||
└── docs/
|
└── docs/
|
||||||
├── API.md # MLS Grid API reference
|
├── API.md # MLS Grid API reference
|
||||||
├── CLAUDE.md # AI assistant context
|
├── CLAUDE.md # AI assistant context
|
||||||
|
|||||||
@@ -512,6 +512,45 @@ class MLS_CLI {
|
|||||||
if (!$result['success']) {
|
if (!$result['success']) {
|
||||||
WP_CLI::halt(1);
|
WP_CLI::halt(1);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Run garbage collection after successful sync
|
||||||
|
if ($result['success']) {
|
||||||
|
$gc = $this->plugin->get_garbage_collector();
|
||||||
|
if ($gc && $gc->is_enabled()) {
|
||||||
|
if (!$silent) {
|
||||||
|
WP_CLI::line('');
|
||||||
|
WP_CLI::line('=== MLS Image Garbage Collection ===');
|
||||||
|
WP_CLI::line('');
|
||||||
|
}
|
||||||
|
|
||||||
|
$gc_result = $gc->run($status_callback);
|
||||||
|
|
||||||
|
if (!$silent && $gc_result['ran']) {
|
||||||
|
WP_CLI::line(sprintf(
|
||||||
|
'Deleted: %d directories (%s)',
|
||||||
|
$gc_result['deleted_count'],
|
||||||
|
$this->format_bytes($gc_result['deleted_bytes'])
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Format bytes to human readable string
|
||||||
|
*
|
||||||
|
* @param int $bytes Bytes
|
||||||
|
* @return string Formatted string
|
||||||
|
*/
|
||||||
|
private function format_bytes($bytes) {
|
||||||
|
if ($bytes >= 1073741824) {
|
||||||
|
return number_format($bytes / 1073741824, 2) . ' GB';
|
||||||
|
} elseif ($bytes >= 1048576) {
|
||||||
|
return number_format($bytes / 1048576, 2) . ' MB';
|
||||||
|
} elseif ($bytes >= 1024) {
|
||||||
|
return number_format($bytes / 1024, 2) . ' KB';
|
||||||
|
}
|
||||||
|
return $bytes . ' bytes';
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|||||||
@@ -0,0 +1,454 @@
|
|||||||
|
<?php
|
||||||
|
/**
|
||||||
|
* MLS Garbage Collector
|
||||||
|
*
|
||||||
|
* Cleans up old MLS image directories when disk space is low.
|
||||||
|
* Runs after sync to prevent disk from filling up.
|
||||||
|
*
|
||||||
|
* Configuration (wp-config.php):
|
||||||
|
* - MLS_GC_DISK_THRESHOLD: Minimum free disk space in bytes before cleanup triggers
|
||||||
|
* Example: define('MLS_GC_DISK_THRESHOLD', 5 * 1024 * 1024 * 1024); // 5GB
|
||||||
|
*
|
||||||
|
* Behavior:
|
||||||
|
* - Only runs if MLS_GC_DISK_THRESHOLD is defined
|
||||||
|
* - Skips directories modified within the last 24 hours
|
||||||
|
* - Deletes oldest directories first
|
||||||
|
* - Stops when free space >= 5GB or 2GB deleted per run
|
||||||
|
*/
|
||||||
|
|
||||||
|
if (!defined('ABSPATH')) {
|
||||||
|
exit;
|
||||||
|
}
|
||||||
|
|
||||||
|
class MLS_Garbage_Collector {
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Target free space to achieve (5GB)
|
||||||
|
*/
|
||||||
|
const TARGET_FREE_SPACE = 5368709120; // 5 * 1024 * 1024 * 1024
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Maximum bytes to delete per run (2GB)
|
||||||
|
*/
|
||||||
|
const MAX_DELETE_PER_RUN = 2147483648; // 2 * 1024 * 1024 * 1024
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Minimum age of directories to delete (24 hours)
|
||||||
|
*/
|
||||||
|
const MIN_AGE_SECONDS = 86400; // 24 * 60 * 60
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Logger instance
|
||||||
|
*/
|
||||||
|
private $logger;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Constructor
|
||||||
|
*
|
||||||
|
* @param MLS_Logger $logger Logger instance
|
||||||
|
*/
|
||||||
|
public function __construct(MLS_Logger $logger) {
|
||||||
|
$this->logger = $logger;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Check if garbage collection is enabled
|
||||||
|
*
|
||||||
|
* @return bool True if MLS_GC_DISK_THRESHOLD is defined
|
||||||
|
*/
|
||||||
|
public function is_enabled() {
|
||||||
|
return defined('MLS_GC_DISK_THRESHOLD') && MLS_GC_DISK_THRESHOLD > 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get the disk threshold from config
|
||||||
|
*
|
||||||
|
* @return int Threshold in bytes, or 0 if not defined
|
||||||
|
*/
|
||||||
|
public function get_threshold() {
|
||||||
|
return defined('MLS_GC_DISK_THRESHOLD') ? (int) MLS_GC_DISK_THRESHOLD : 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get the MLS images upload directory
|
||||||
|
*
|
||||||
|
* @return string Absolute path to MLS images directory
|
||||||
|
*/
|
||||||
|
public function get_images_dir() {
|
||||||
|
$upload_dir = wp_upload_dir();
|
||||||
|
return $upload_dir['basedir'] . '/mls-listings';
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get free disk space for the volume hosting MLS images
|
||||||
|
*
|
||||||
|
* @return int|false Free space in bytes, or false on error
|
||||||
|
*/
|
||||||
|
public function get_free_space() {
|
||||||
|
$dir = $this->get_images_dir();
|
||||||
|
|
||||||
|
// Create directory if it doesn't exist
|
||||||
|
if (!file_exists($dir)) {
|
||||||
|
wp_mkdir_p($dir);
|
||||||
|
}
|
||||||
|
|
||||||
|
return disk_free_space($dir);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Check if cleanup is needed
|
||||||
|
*
|
||||||
|
* @return bool True if free space is below threshold
|
||||||
|
*/
|
||||||
|
public function needs_cleanup() {
|
||||||
|
if (!$this->is_enabled()) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
$free_space = $this->get_free_space();
|
||||||
|
if ($free_space === false) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
return $free_space < $this->get_threshold();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get listing directories sorted by modification time (oldest first)
|
||||||
|
*
|
||||||
|
* Only returns directories older than MIN_AGE_SECONDS.
|
||||||
|
*
|
||||||
|
* @return array Array of ['path' => string, 'mtime' => int, 'size' => int]
|
||||||
|
*/
|
||||||
|
public function get_old_directories() {
|
||||||
|
$base_dir = $this->get_images_dir();
|
||||||
|
$cutoff_time = time() - self::MIN_AGE_SECONDS;
|
||||||
|
$directories = array();
|
||||||
|
|
||||||
|
if (!is_dir($base_dir)) {
|
||||||
|
return $directories;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Iterate through prefix directories (2-char subdirs like "AB", "CD")
|
||||||
|
$prefix_dirs = glob($base_dir . '/*', GLOB_ONLYDIR);
|
||||||
|
if (!$prefix_dirs) {
|
||||||
|
return $directories;
|
||||||
|
}
|
||||||
|
|
||||||
|
foreach ($prefix_dirs as $prefix_dir) {
|
||||||
|
// Iterate through listing directories within each prefix
|
||||||
|
$listing_dirs = glob($prefix_dir . '/*', GLOB_ONLYDIR);
|
||||||
|
if (!$listing_dirs) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
foreach ($listing_dirs as $listing_dir) {
|
||||||
|
$mtime = filemtime($listing_dir);
|
||||||
|
|
||||||
|
// Skip if modified within the last 24 hours
|
||||||
|
if ($mtime >= $cutoff_time) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
$directories[] = array(
|
||||||
|
'path' => $listing_dir,
|
||||||
|
'mtime' => $mtime,
|
||||||
|
'size' => $this->get_directory_size($listing_dir),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Sort by modification time (oldest first)
|
||||||
|
usort($directories, function($a, $b) {
|
||||||
|
return $a['mtime'] - $b['mtime'];
|
||||||
|
});
|
||||||
|
|
||||||
|
return $directories;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get total size of a directory
|
||||||
|
*
|
||||||
|
* @param string $dir Directory path
|
||||||
|
* @return int Size in bytes
|
||||||
|
*/
|
||||||
|
private function get_directory_size($dir) {
|
||||||
|
$size = 0;
|
||||||
|
$files = new RecursiveIteratorIterator(
|
||||||
|
new RecursiveDirectoryIterator($dir, RecursiveDirectoryIterator::SKIP_DOTS),
|
||||||
|
RecursiveIteratorIterator::LEAVES_ONLY
|
||||||
|
);
|
||||||
|
|
||||||
|
foreach ($files as $file) {
|
||||||
|
if ($file->isFile()) {
|
||||||
|
$size += $file->getSize();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return $size;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Delete a directory and all its contents
|
||||||
|
*
|
||||||
|
* @param string $dir Directory path
|
||||||
|
* @return bool True on success
|
||||||
|
*/
|
||||||
|
private function delete_directory($dir) {
|
||||||
|
if (!is_dir($dir)) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
$files = new RecursiveIteratorIterator(
|
||||||
|
new RecursiveDirectoryIterator($dir, RecursiveDirectoryIterator::SKIP_DOTS),
|
||||||
|
RecursiveIteratorIterator::CHILD_FIRST
|
||||||
|
);
|
||||||
|
|
||||||
|
foreach ($files as $file) {
|
||||||
|
if ($file->isDir()) {
|
||||||
|
@rmdir($file->getRealPath());
|
||||||
|
} else {
|
||||||
|
@unlink($file->getRealPath());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return @rmdir($dir);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Run garbage collection
|
||||||
|
*
|
||||||
|
* Deletes old directories until free space >= TARGET_FREE_SPACE
|
||||||
|
* or MAX_DELETE_PER_RUN bytes have been deleted.
|
||||||
|
*
|
||||||
|
* @param callable|null $status_callback Optional callback for status messages
|
||||||
|
* @return array Results with 'deleted_count', 'deleted_bytes', 'free_space_before', 'free_space_after'
|
||||||
|
*/
|
||||||
|
public function run($status_callback = null) {
|
||||||
|
$result = array(
|
||||||
|
'enabled' => $this->is_enabled(),
|
||||||
|
'ran' => false,
|
||||||
|
'deleted_count' => 0,
|
||||||
|
'deleted_bytes' => 0,
|
||||||
|
'free_space_before' => 0,
|
||||||
|
'free_space_after' => 0,
|
||||||
|
'threshold' => $this->get_threshold(),
|
||||||
|
);
|
||||||
|
|
||||||
|
// Check if enabled
|
||||||
|
if (!$this->is_enabled()) {
|
||||||
|
if ($status_callback) {
|
||||||
|
$status_callback('Garbage collection disabled (MLS_GC_DISK_THRESHOLD not defined)', 'info');
|
||||||
|
}
|
||||||
|
return $result;
|
||||||
|
}
|
||||||
|
|
||||||
|
$free_space = $this->get_free_space();
|
||||||
|
if ($free_space === false) {
|
||||||
|
if ($status_callback) {
|
||||||
|
$status_callback('Could not determine free disk space', 'warning');
|
||||||
|
}
|
||||||
|
return $result;
|
||||||
|
}
|
||||||
|
|
||||||
|
$result['free_space_before'] = $free_space;
|
||||||
|
$threshold = $this->get_threshold();
|
||||||
|
|
||||||
|
// Check if cleanup needed
|
||||||
|
if ($free_space >= $threshold) {
|
||||||
|
if ($status_callback) {
|
||||||
|
$status_callback(sprintf(
|
||||||
|
'Disk space OK: %s free (threshold: %s)',
|
||||||
|
$this->format_bytes($free_space),
|
||||||
|
$this->format_bytes($threshold)
|
||||||
|
), 'info');
|
||||||
|
}
|
||||||
|
$result['free_space_after'] = $free_space;
|
||||||
|
return $result;
|
||||||
|
}
|
||||||
|
|
||||||
|
$result['ran'] = true;
|
||||||
|
|
||||||
|
if ($status_callback) {
|
||||||
|
$status_callback(sprintf(
|
||||||
|
'Disk space low: %s free (threshold: %s). Starting cleanup...',
|
||||||
|
$this->format_bytes($free_space),
|
||||||
|
$this->format_bytes($threshold)
|
||||||
|
), 'warning');
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get old directories
|
||||||
|
$directories = $this->get_old_directories();
|
||||||
|
|
||||||
|
if (empty($directories)) {
|
||||||
|
if ($status_callback) {
|
||||||
|
$status_callback('No directories older than 24 hours found for cleanup', 'info');
|
||||||
|
}
|
||||||
|
$result['free_space_after'] = $free_space;
|
||||||
|
return $result;
|
||||||
|
}
|
||||||
|
|
||||||
|
$deleted_bytes = 0;
|
||||||
|
$deleted_count = 0;
|
||||||
|
|
||||||
|
foreach ($directories as $dir_info) {
|
||||||
|
// Stop if we've reached target free space
|
||||||
|
$current_free = $this->get_free_space();
|
||||||
|
if ($current_free !== false && $current_free >= self::TARGET_FREE_SPACE) {
|
||||||
|
if ($status_callback) {
|
||||||
|
$status_callback(sprintf(
|
||||||
|
'Target free space reached: %s',
|
||||||
|
$this->format_bytes($current_free)
|
||||||
|
), 'info');
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stop if we've deleted enough this run
|
||||||
|
if ($deleted_bytes >= self::MAX_DELETE_PER_RUN) {
|
||||||
|
if ($status_callback) {
|
||||||
|
$status_callback(sprintf(
|
||||||
|
'Max deletion limit reached: %s',
|
||||||
|
$this->format_bytes($deleted_bytes)
|
||||||
|
), 'info');
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Delete the directory
|
||||||
|
$path = $dir_info['path'];
|
||||||
|
$size = $dir_info['size'];
|
||||||
|
$listing_key = basename($path);
|
||||||
|
|
||||||
|
if ($this->delete_directory($path)) {
|
||||||
|
$deleted_bytes += $size;
|
||||||
|
$deleted_count++;
|
||||||
|
|
||||||
|
$this->logger->info('Garbage collection deleted directory', array(
|
||||||
|
'listing_key' => $listing_key,
|
||||||
|
'size' => $size,
|
||||||
|
'age_days' => round((time() - $dir_info['mtime']) / 86400, 1),
|
||||||
|
));
|
||||||
|
|
||||||
|
if ($status_callback) {
|
||||||
|
$status_callback(sprintf(
|
||||||
|
'Deleted: %s (%s)',
|
||||||
|
$listing_key,
|
||||||
|
$this->format_bytes($size)
|
||||||
|
), 'info');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Clean up empty prefix directories
|
||||||
|
$this->cleanup_empty_prefix_dirs();
|
||||||
|
|
||||||
|
$result['deleted_count'] = $deleted_count;
|
||||||
|
$result['deleted_bytes'] = $deleted_bytes;
|
||||||
|
$result['free_space_after'] = $this->get_free_space();
|
||||||
|
|
||||||
|
if ($status_callback) {
|
||||||
|
$status_callback(sprintf(
|
||||||
|
'Cleanup complete: Deleted %d directories (%s). Free space now: %s',
|
||||||
|
$deleted_count,
|
||||||
|
$this->format_bytes($deleted_bytes),
|
||||||
|
$this->format_bytes($result['free_space_after'])
|
||||||
|
), 'info');
|
||||||
|
}
|
||||||
|
|
||||||
|
$this->logger->info('Garbage collection completed', array(
|
||||||
|
'deleted_count' => $deleted_count,
|
||||||
|
'deleted_bytes' => $deleted_bytes,
|
||||||
|
'free_space_before' => $result['free_space_before'],
|
||||||
|
'free_space_after' => $result['free_space_after'],
|
||||||
|
));
|
||||||
|
|
||||||
|
return $result;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Clean up empty prefix directories
|
||||||
|
*/
|
||||||
|
private function cleanup_empty_prefix_dirs() {
|
||||||
|
$base_dir = $this->get_images_dir();
|
||||||
|
$prefix_dirs = glob($base_dir . '/*', GLOB_ONLYDIR);
|
||||||
|
|
||||||
|
if (!$prefix_dirs) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
foreach ($prefix_dirs as $prefix_dir) {
|
||||||
|
// Check if directory is empty
|
||||||
|
$contents = glob($prefix_dir . '/*');
|
||||||
|
if (empty($contents)) {
|
||||||
|
@rmdir($prefix_dir);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Format bytes to human readable string
|
||||||
|
*
|
||||||
|
* @param int $bytes Bytes
|
||||||
|
* @return string Formatted string (e.g., "1.5 GB")
|
||||||
|
*/
|
||||||
|
private function format_bytes($bytes) {
|
||||||
|
if ($bytes >= 1073741824) {
|
||||||
|
return number_format($bytes / 1073741824, 2) . ' GB';
|
||||||
|
} elseif ($bytes >= 1048576) {
|
||||||
|
return number_format($bytes / 1048576, 2) . ' MB';
|
||||||
|
} elseif ($bytes >= 1024) {
|
||||||
|
return number_format($bytes / 1024, 2) . ' KB';
|
||||||
|
}
|
||||||
|
return $bytes . ' bytes';
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get statistics about the image cache
|
||||||
|
*
|
||||||
|
* @return array Stats including total_size, directory_count, oldest_mtime
|
||||||
|
*/
|
||||||
|
public function get_stats() {
|
||||||
|
$base_dir = $this->get_images_dir();
|
||||||
|
$stats = array(
|
||||||
|
'total_size' => 0,
|
||||||
|
'directory_count' => 0,
|
||||||
|
'oldest_mtime' => null,
|
||||||
|
'newest_mtime' => null,
|
||||||
|
'free_space' => $this->get_free_space(),
|
||||||
|
'threshold' => $this->get_threshold(),
|
||||||
|
'needs_cleanup' => $this->needs_cleanup(),
|
||||||
|
);
|
||||||
|
|
||||||
|
if (!is_dir($base_dir)) {
|
||||||
|
return $stats;
|
||||||
|
}
|
||||||
|
|
||||||
|
$prefix_dirs = glob($base_dir . '/*', GLOB_ONLYDIR);
|
||||||
|
if (!$prefix_dirs) {
|
||||||
|
return $stats;
|
||||||
|
}
|
||||||
|
|
||||||
|
foreach ($prefix_dirs as $prefix_dir) {
|
||||||
|
$listing_dirs = glob($prefix_dir . '/*', GLOB_ONLYDIR);
|
||||||
|
if (!$listing_dirs) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
foreach ($listing_dirs as $listing_dir) {
|
||||||
|
$stats['directory_count']++;
|
||||||
|
$stats['total_size'] += $this->get_directory_size($listing_dir);
|
||||||
|
|
||||||
|
$mtime = filemtime($listing_dir);
|
||||||
|
if ($stats['oldest_mtime'] === null || $mtime < $stats['oldest_mtime']) {
|
||||||
|
$stats['oldest_mtime'] = $mtime;
|
||||||
|
}
|
||||||
|
if ($stats['newest_mtime'] === null || $mtime > $stats['newest_mtime']) {
|
||||||
|
$stats['newest_mtime'] = $mtime;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return $stats;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -39,6 +39,127 @@ class MLS_Media_Handler {
|
|||||||
$this->logger = $logger;
|
$this->logger = $logger;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* WebP quality setting (80 is roughly equivalent to JPEG 90)
|
||||||
|
*/
|
||||||
|
const WEBP_QUALITY = 80;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Max image width in pixels
|
||||||
|
*/
|
||||||
|
const MAX_IMAGE_WIDTH = 1600;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* File size threshold for WebP conversion (500KB)
|
||||||
|
*/
|
||||||
|
const WEBP_SIZE_THRESHOLD = 512000;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Convert image to WebP format if needed
|
||||||
|
*
|
||||||
|
* Converts to WebP if:
|
||||||
|
* - Image is PNG, OR
|
||||||
|
* - Image is larger than WEBP_SIZE_THRESHOLD
|
||||||
|
*
|
||||||
|
* Also resizes if width > MAX_IMAGE_WIDTH
|
||||||
|
*
|
||||||
|
* @param string $file_path Absolute path to image file
|
||||||
|
* @param string $extension Original file extension
|
||||||
|
* @return array ['path' => new path, 'extension' => new extension, 'converted' => bool]
|
||||||
|
*/
|
||||||
|
private function maybe_convert_to_webp($file_path, $extension) {
|
||||||
|
$result = array(
|
||||||
|
'path' => $file_path,
|
||||||
|
'extension' => $extension,
|
||||||
|
'converted' => false,
|
||||||
|
);
|
||||||
|
|
||||||
|
// Skip if already WebP or GIF (preserve animations)
|
||||||
|
if ($extension === 'webp' || $extension === 'gif') {
|
||||||
|
return $result;
|
||||||
|
}
|
||||||
|
|
||||||
|
$file_size = filesize($file_path);
|
||||||
|
$is_png = ($extension === 'png');
|
||||||
|
$is_large = ($file_size > self::WEBP_SIZE_THRESHOLD);
|
||||||
|
|
||||||
|
// Only convert PNGs or large files
|
||||||
|
if (!$is_png && !$is_large) {
|
||||||
|
return $result;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check if we have image editing capability
|
||||||
|
if (!function_exists('wp_get_image_editor')) {
|
||||||
|
return $result;
|
||||||
|
}
|
||||||
|
|
||||||
|
$editor = wp_get_image_editor($file_path);
|
||||||
|
if (is_wp_error($editor)) {
|
||||||
|
$this->logger->warning('Could not load image for WebP conversion', array(
|
||||||
|
'path' => $file_path,
|
||||||
|
'error' => $editor->get_error_message(),
|
||||||
|
));
|
||||||
|
return $result;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get current dimensions
|
||||||
|
$size = $editor->get_size();
|
||||||
|
$needs_resize = ($size['width'] > self::MAX_IMAGE_WIDTH);
|
||||||
|
|
||||||
|
// Resize if needed
|
||||||
|
if ($needs_resize) {
|
||||||
|
$editor->resize(self::MAX_IMAGE_WIDTH, null, false);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Set quality (80 is roughly equivalent to JPEG 90)
|
||||||
|
$editor->set_quality(self::WEBP_QUALITY);
|
||||||
|
|
||||||
|
// Generate WebP path
|
||||||
|
$webp_path = preg_replace('/\.[^.]+$/', '.webp', $file_path);
|
||||||
|
|
||||||
|
// Save as WebP
|
||||||
|
$saved = $editor->save($webp_path, 'image/webp');
|
||||||
|
|
||||||
|
if (is_wp_error($saved)) {
|
||||||
|
$this->logger->warning('WebP conversion failed', array(
|
||||||
|
'path' => $file_path,
|
||||||
|
'error' => $saved->get_error_message(),
|
||||||
|
));
|
||||||
|
return $result;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Delete original file
|
||||||
|
@unlink($file_path);
|
||||||
|
|
||||||
|
$this->logger->debug('Converted image to WebP', array(
|
||||||
|
'original_path' => $file_path,
|
||||||
|
'original_size' => $file_size,
|
||||||
|
'webp_path' => $webp_path,
|
||||||
|
'webp_size' => filesize($webp_path),
|
||||||
|
'resized' => $needs_resize,
|
||||||
|
));
|
||||||
|
|
||||||
|
return array(
|
||||||
|
'path' => $webp_path,
|
||||||
|
'extension' => 'webp',
|
||||||
|
'converted' => true,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Check if WebP version of file exists, return that path if so
|
||||||
|
*
|
||||||
|
* @param string $file_path Original file path
|
||||||
|
* @return string Path to use (WebP if exists, otherwise original)
|
||||||
|
*/
|
||||||
|
private function prefer_webp_path($file_path) {
|
||||||
|
$webp_path = preg_replace('/\.[^.]+$/', '.webp', $file_path);
|
||||||
|
if (file_exists($webp_path)) {
|
||||||
|
return $webp_path;
|
||||||
|
}
|
||||||
|
return $file_path;
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Get base upload directory for MLS media
|
* Get base upload directory for MLS media
|
||||||
*
|
*
|
||||||
@@ -195,10 +316,16 @@ class MLS_Media_Handler {
|
|||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Already cached
|
// Already cached - check for WebP version first
|
||||||
if ($media->local_url && $media->local_path) {
|
if ($media->local_url && $media->local_path) {
|
||||||
$file_path = $this->get_upload_dir() . '/' . $media->local_path;
|
$file_path = $this->get_upload_dir() . '/' . $media->local_path;
|
||||||
if (file_exists($file_path)) {
|
$actual_path = $this->prefer_webp_path($file_path);
|
||||||
|
if (file_exists($actual_path)) {
|
||||||
|
// If WebP version exists, return WebP URL
|
||||||
|
if ($actual_path !== $file_path) {
|
||||||
|
$webp_path = preg_replace('/\.[^.]+$/', '.webp', $media->local_path);
|
||||||
|
return $this->get_upload_url() . '/' . $webp_path;
|
||||||
|
}
|
||||||
return $media->local_url;
|
return $media->local_url;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -235,7 +362,13 @@ class MLS_Media_Handler {
|
|||||||
|
|
||||||
if ($cached) {
|
if ($cached) {
|
||||||
$file_path = $this->get_upload_dir() . '/' . $cached->local_path;
|
$file_path = $this->get_upload_dir() . '/' . $cached->local_path;
|
||||||
if (file_exists($file_path)) {
|
$actual_path = $this->prefer_webp_path($file_path);
|
||||||
|
if (file_exists($actual_path)) {
|
||||||
|
// If WebP version exists, return WebP URL
|
||||||
|
if ($actual_path !== $file_path) {
|
||||||
|
$webp_path = preg_replace('/\.[^.]+$/', '.webp', $cached->local_path);
|
||||||
|
return $this->get_upload_url() . '/' . $webp_path;
|
||||||
|
}
|
||||||
return $cached->local_url;
|
return $cached->local_url;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -253,10 +386,16 @@ class MLS_Media_Handler {
|
|||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
// If already cached and file exists, return it
|
// If already cached and file exists, return it - check for WebP first
|
||||||
if ($media->local_url && $media->local_path) {
|
if ($media->local_url && $media->local_path) {
|
||||||
$file_path = $this->get_upload_dir() . '/' . $media->local_path;
|
$file_path = $this->get_upload_dir() . '/' . $media->local_path;
|
||||||
if (file_exists($file_path)) {
|
$actual_path = $this->prefer_webp_path($file_path);
|
||||||
|
if (file_exists($actual_path)) {
|
||||||
|
// If WebP version exists, return WebP URL
|
||||||
|
if ($actual_path !== $file_path) {
|
||||||
|
$webp_path = preg_replace('/\.[^.]+$/', '.webp', $media->local_path);
|
||||||
|
return $this->get_upload_url() . '/' . $webp_path;
|
||||||
|
}
|
||||||
return $media->local_url;
|
return $media->local_url;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -292,10 +431,16 @@ class MLS_Media_Handler {
|
|||||||
|
|
||||||
$fetched = 0;
|
$fetched = 0;
|
||||||
foreach ($media as &$item) {
|
foreach ($media as &$item) {
|
||||||
// Check if cached and file exists
|
// Check if cached and file exists - prefer WebP version
|
||||||
if ($item->local_url && $item->local_path) {
|
if ($item->local_url && $item->local_path) {
|
||||||
$file_path = $this->get_upload_dir() . '/' . $item->local_path;
|
$file_path = $this->get_upload_dir() . '/' . $item->local_path;
|
||||||
if (file_exists($file_path)) {
|
$actual_path = $this->prefer_webp_path($file_path);
|
||||||
|
if (file_exists($actual_path)) {
|
||||||
|
// If WebP version exists, update the URL
|
||||||
|
if ($actual_path !== $file_path) {
|
||||||
|
$webp_path = preg_replace('/\.[^.]+$/', '.webp', $item->local_path);
|
||||||
|
$item->local_url = $this->get_upload_url() . '/' . $webp_path;
|
||||||
|
}
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -358,8 +503,14 @@ class MLS_Media_Handler {
|
|||||||
|
|
||||||
if ($updated_media && $updated_media->local_path) {
|
if ($updated_media && $updated_media->local_path) {
|
||||||
$file_path = $this->get_upload_dir() . '/' . $updated_media->local_path;
|
$file_path = $this->get_upload_dir() . '/' . $updated_media->local_path;
|
||||||
if (file_exists($file_path)) {
|
$actual_path = $this->prefer_webp_path($file_path);
|
||||||
|
if (file_exists($actual_path)) {
|
||||||
// Another request cached it while we waited
|
// Another request cached it while we waited
|
||||||
|
// If WebP version exists, return WebP URL
|
||||||
|
if ($actual_path !== $file_path) {
|
||||||
|
$webp_path = preg_replace('/\.[^.]+$/', '.webp', $updated_media->local_path);
|
||||||
|
return $this->get_upload_url() . '/' . $webp_path;
|
||||||
|
}
|
||||||
return $updated_media->local_url;
|
return $updated_media->local_url;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -415,17 +566,29 @@ class MLS_Media_Handler {
|
|||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Convert to WebP if PNG or file is large (>500KB)
|
||||||
|
$conversion = $this->maybe_convert_to_webp($file_path, $extension);
|
||||||
|
if ($conversion['converted']) {
|
||||||
|
$file_path = $conversion['path'];
|
||||||
|
$extension = $conversion['extension'];
|
||||||
|
$filename = $media->media_order . '.' . $extension;
|
||||||
|
$content_type = 'image/webp';
|
||||||
|
}
|
||||||
|
|
||||||
// Update database
|
// Update database
|
||||||
$prefix = substr($media->listing_key, 0, 2);
|
$prefix = substr($media->listing_key, 0, 2);
|
||||||
$relative_path = $prefix . '/' . $media->listing_key . '/' . $filename;
|
$relative_path = $prefix . '/' . $media->listing_key . '/' . $filename;
|
||||||
$local_url = $this->get_upload_url() . '/' . $relative_path;
|
$local_url = $this->get_upload_url() . '/' . $relative_path;
|
||||||
|
|
||||||
|
// Get actual file size after any conversion
|
||||||
|
$final_size = filesize($file_path);
|
||||||
|
|
||||||
$wpdb->update(
|
$wpdb->update(
|
||||||
$this->db->media_table(),
|
$this->db->media_table(),
|
||||||
array(
|
array(
|
||||||
'local_path' => $relative_path,
|
'local_path' => $relative_path,
|
||||||
'local_url' => $local_url,
|
'local_url' => $local_url,
|
||||||
'file_size' => strlen($body),
|
'file_size' => $final_size,
|
||||||
'mime_type' => $content_type,
|
'mime_type' => $content_type,
|
||||||
'downloaded_at' => current_time('mysql'),
|
'downloaded_at' => current_time('mysql'),
|
||||||
),
|
),
|
||||||
@@ -435,7 +598,9 @@ class MLS_Media_Handler {
|
|||||||
$this->logger->debug('Media fetched and cached', array(
|
$this->logger->debug('Media fetched and cached', array(
|
||||||
'listing_key' => $media->listing_key,
|
'listing_key' => $media->listing_key,
|
||||||
'media_key' => $media->media_key,
|
'media_key' => $media->media_key,
|
||||||
'size' => strlen($body),
|
'original_size' => strlen($body),
|
||||||
|
'final_size' => $final_size,
|
||||||
|
'converted' => $conversion['converted'],
|
||||||
));
|
));
|
||||||
|
|
||||||
return $local_url;
|
return $local_url;
|
||||||
|
|||||||
@@ -63,6 +63,7 @@ final class MLS_Plugin {
|
|||||||
private $image_endpoint;
|
private $image_endpoint;
|
||||||
private $query;
|
private $query;
|
||||||
private $cluster;
|
private $cluster;
|
||||||
|
private $garbage_collector;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Get single instance
|
* Get single instance
|
||||||
@@ -98,6 +99,7 @@ final class MLS_Plugin {
|
|||||||
require_once MLS_PLUGIN_DIR . 'includes/class-mls-query.php';
|
require_once MLS_PLUGIN_DIR . 'includes/class-mls-query.php';
|
||||||
require_once MLS_PLUGIN_DIR . 'includes/class-mls-cluster.php';
|
require_once MLS_PLUGIN_DIR . 'includes/class-mls-cluster.php';
|
||||||
require_once MLS_PLUGIN_DIR . 'includes/class-mls-geo-validator.php';
|
require_once MLS_PLUGIN_DIR . 'includes/class-mls-geo-validator.php';
|
||||||
|
require_once MLS_PLUGIN_DIR . 'includes/class-mls-garbage-collector.php';
|
||||||
|
|
||||||
// Activation/Deactivation
|
// Activation/Deactivation
|
||||||
require_once MLS_PLUGIN_DIR . 'includes/class-mls-activator.php';
|
require_once MLS_PLUGIN_DIR . 'includes/class-mls-activator.php';
|
||||||
@@ -149,6 +151,7 @@ final class MLS_Plugin {
|
|||||||
);
|
);
|
||||||
$this->query = new MLS_Query($this->db);
|
$this->query = new MLS_Query($this->db);
|
||||||
$this->cluster = new MLS_Cluster($this->db);
|
$this->cluster = new MLS_Cluster($this->db);
|
||||||
|
$this->garbage_collector = new MLS_Garbage_Collector($this->logger);
|
||||||
|
|
||||||
// Register AJAX handlers
|
// Register AJAX handlers
|
||||||
add_action('wp_ajax_mls_get_clusters', array($this, 'ajax_get_clusters'));
|
add_action('wp_ajax_mls_get_clusters', array($this, 'ajax_get_clusters'));
|
||||||
@@ -247,6 +250,13 @@ final class MLS_Plugin {
|
|||||||
return $this->cluster;
|
return $this->cluster;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Get Garbage Collector instance
|
||||||
|
*/
|
||||||
|
public function get_garbage_collector() {
|
||||||
|
return $this->garbage_collector;
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* AJAX handler for getting map clusters
|
* AJAX handler for getting map clusters
|
||||||
*/
|
*/
|
||||||
|
|||||||
Reference in New Issue
Block a user