feat: Add VPS storage system and complete integration
🎯 Overview: Implemented complete VPS-based storage system allowing the iOS app to download and store manga chapters on the VPS for ad-free offline reading. 📦 Backend Changes: - Added storage.js service for managing chapter downloads (270 lines) - Updated server.js with 6 new storage endpoints: - POST /api/download - Download chapters to VPS - GET /api/storage/chapters/:mangaSlug - List downloaded chapters - GET /api/storage/chapter/:mangaSlug/:chapterNumber - Check download status - GET /api/storage/image/:mangaSlug/:chapterNumber/:pageIndex - Serve images - DELETE /api/storage/chapter/:mangaSlug/:chapterNumber - Delete chapters - GET /api/storage/stats - Get storage statistics - Fixed scraper.js Puppeteer compatibility issues (waitForTimeout, networkidle0) - Added comprehensive test suite: - test-vps-flow.js (13 tests - 100% pass rate) - test-concurrent-downloads.js (10 tests for parallel operations) - run-tests.sh automation script 📱 iOS App Changes: - Created APIConfig.swift with VPS connection settings - Created VPSAPIClient.swift service (727 lines) for backend communication - Updated MangaDetailView.swift with VPS download integration: - Cloud icon for VPS-available chapters - Upload button to download chapters to VPS - Progress indicators for active downloads - Bulk download options (last 10 or all chapters) - Updated ReaderView.swift to load images from VPS first - Progressive enhancement: app works without VPS, enhances when available ✅ Tests: - All 13 VPS flow tests passing (100%) - Tests verify: scraping, downloading, storage, serving, deletion, stats - Chapter 789 download test: 21 images, 4.68 MB - Concurrent download tests verify no race conditions 🔧 Configuration: - VPS URL: https://gitea.cbcren.online:3001 - Storage location: /home/ren/ios/MangaReader/storage/ - Static file serving: /storage path 📚 Documentation: - Added VPS_INTEGRATION_SUMMARY.md - Complete feature overview - Added CHANGES.md - Detailed code changes reference - Added TEST_README.md, TEST_QUICK_START.md, TEST_SUMMARY.md - Added APIConfig README with usage examples 🤖 Generated with Claude Code (https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
246
backend/TEST_README.md
Normal file
246
backend/TEST_README.md
Normal file
@@ -0,0 +1,246 @@
|
||||
# Integration Tests for MangaReader VPS Backend
|
||||
|
||||
This directory contains comprehensive integration tests for the complete VPS flow: iOS app → VPS backend → Storage → Images served back.
|
||||
|
||||
## Test Files
|
||||
|
||||
### 1. `test-vps-flow.js`
|
||||
Tests the complete end-to-end flow of downloading and serving manga chapters.
|
||||
|
||||
**Test Coverage:**
|
||||
- Server health check
|
||||
- Chapter image scraping from source
|
||||
- Download to VPS storage
|
||||
- Storage verification
|
||||
- Image file validation
|
||||
- Image path retrieval
|
||||
- Chapter listing
|
||||
- Storage statistics
|
||||
- Chapter deletion
|
||||
- Post-deletion verification
|
||||
- Storage stats update verification
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
# Make sure the server is running first
|
||||
node server.js &
|
||||
|
||||
# In another terminal, run the test
|
||||
node test-vps-flow.js
|
||||
```
|
||||
|
||||
**Expected Output:**
|
||||
- Color-coded test progress
|
||||
- Detailed assertions with success/failure indicators
|
||||
- Storage statistics
|
||||
- Final summary with pass/fail counts
|
||||
|
||||
### 2. `test-concurrent-downloads.js`
|
||||
Tests concurrent download operations to verify thread safety and data integrity.
|
||||
|
||||
**Test Coverage:**
|
||||
- Pre-download cleanup
|
||||
- Concurrent chapter downloads (5 chapters, max 3 concurrent)
|
||||
- Post-download verification
|
||||
- File integrity checks (no corruption, no missing files)
|
||||
- Manifest independence verification
|
||||
- Storage statistics accuracy
|
||||
- Chapter listing functionality
|
||||
- Concurrent deletion
|
||||
- Complete cleanup verification
|
||||
- Race condition detection
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
# Make sure the server is running first
|
||||
node server.js &
|
||||
|
||||
# In another terminal, run the test
|
||||
node test-concurrent-downloads.js
|
||||
```
|
||||
|
||||
**Expected Output:**
|
||||
- Progress tracking for each operation
|
||||
- Batch processing information
|
||||
- Detailed integrity reports per chapter
|
||||
- Summary of valid/missing/corrupted images
|
||||
- Concurrent delete tracking
|
||||
- Final summary with race condition analysis
|
||||
|
||||
## Test Configuration
|
||||
|
||||
Both tests use the following configuration:
|
||||
|
||||
```javascript
|
||||
{
|
||||
mangaSlug: 'one-piece_1695365223767',
|
||||
chapters: [787, 788, 789, 790, 791], // For concurrent test
|
||||
baseUrl: 'http://localhost:3000',
|
||||
timeout: 120000-180000 // 2-3 minutes
|
||||
}
|
||||
```
|
||||
|
||||
You can modify these values in the test files if needed.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. **Dependencies installed:**
|
||||
```bash
|
||||
npm install
|
||||
```
|
||||
|
||||
2. **Server running on port 3000:**
|
||||
```bash
|
||||
node server.js
|
||||
```
|
||||
|
||||
3. **Storage directory structure:**
|
||||
The tests will automatically create the required storage structure:
|
||||
```
|
||||
/storage
|
||||
/manga
|
||||
/one-piece_1695365223767
|
||||
/chapter_789
|
||||
page_001.jpg
|
||||
page_002.jpg
|
||||
...
|
||||
manifest.json
|
||||
```
|
||||
|
||||
## Running All Tests
|
||||
|
||||
Run both test suites:
|
||||
|
||||
```bash
|
||||
# Terminal 1: Start server
|
||||
cd /home/ren/ios/MangaReader/backend
|
||||
node server.js
|
||||
|
||||
# Terminal 2: Run VPS flow test
|
||||
node test-vps-flow.js
|
||||
|
||||
# Terminal 3: Run concurrent downloads test
|
||||
node test-concurrent-downloads.js
|
||||
```
|
||||
|
||||
## Test Results
|
||||
|
||||
### Success Indicators
|
||||
- ✓ Green checkmarks for passing assertions
|
||||
- 🎉 "ALL TESTS PASSED!" message
|
||||
- Exit code 0
|
||||
|
||||
### Failure Indicators
|
||||
- ✗ Red X marks for failing assertions
|
||||
- ❌ "SOME TESTS FAILED" message
|
||||
- Detailed error messages
|
||||
- Exit code 1
|
||||
|
||||
## Color Codes
|
||||
|
||||
The tests use color-coded output for easy reading:
|
||||
- **Green**: Success/passing assertions
|
||||
- **Red**: Errors/failing assertions
|
||||
- **Blue**: Information messages
|
||||
- **Cyan**: Test titles
|
||||
- **Yellow**: Warnings
|
||||
- **Magenta**: Operation tracking (concurrent tests)
|
||||
|
||||
## Cleanup
|
||||
|
||||
Tests automatically clean up after themselves:
|
||||
- Delete test chapters from storage
|
||||
- Remove temporary files
|
||||
- Reset storage statistics
|
||||
|
||||
However, you can manually clean up:
|
||||
|
||||
```bash
|
||||
# Remove all test data
|
||||
rm -rf /home/ren/ios/MangaReader/storage/manga/one-piece_1695365223767
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Server Not Responding
|
||||
```
|
||||
Error: Failed to fetch
|
||||
```
|
||||
**Solution:** Make sure the server is running on port 3000:
|
||||
```bash
|
||||
node server.js
|
||||
```
|
||||
|
||||
### Chapter Already Exists
|
||||
Tests will automatically clean up existing chapters. If you see warnings, that's normal behavior.
|
||||
|
||||
### Timeout Errors
|
||||
If tests timeout, the scraper might be taking too long. You can:
|
||||
1. Increase the timeout value in TEST_CONFIG
|
||||
2. Check your internet connection
|
||||
3. Verify the source website is accessible
|
||||
|
||||
### Port Already in Use
|
||||
```
|
||||
Error: listen EADDRINUSE: address already in use :::3000
|
||||
```
|
||||
**Solution:** Kill the existing process:
|
||||
```bash
|
||||
lsof -ti:3000 | xargs kill -9
|
||||
```
|
||||
|
||||
## Test Coverage Summary
|
||||
|
||||
| Feature | VPS Flow Test | Concurrent Test |
|
||||
|---------|---------------|-----------------|
|
||||
| Server Health | ✓ | - |
|
||||
| Image Scraping | ✓ | ✓ |
|
||||
| Download to Storage | ✓ | ✓ (5 chapters) |
|
||||
| File Verification | ✓ | ✓ |
|
||||
| Manifest Validation | ✓ | ✓ |
|
||||
| Storage Stats | ✓ | ✓ |
|
||||
| Chapter Listing | ✓ | ✓ |
|
||||
| Deletion | ✓ | ✓ (concurrent) |
|
||||
| Race Conditions | - | ✓ |
|
||||
| Corruption Detection | - | ✓ |
|
||||
|
||||
## Integration with CI/CD
|
||||
|
||||
These tests can be integrated into a CI/CD pipeline:
|
||||
|
||||
```yaml
|
||||
# Example GitHub Actions workflow
|
||||
- name: Start Server
|
||||
run: node server.js &
|
||||
|
||||
- name: Wait for Server
|
||||
run: sleep 5
|
||||
|
||||
- name: Run VPS Flow Tests
|
||||
run: node test-vps-flow.js
|
||||
|
||||
- name: Run Concurrent Tests
|
||||
run: node test-concurrent-downloads.js
|
||||
```
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- **VPS Flow Test**: ~2-3 minutes (downloads 5 images from 1 chapter)
|
||||
- **Concurrent Test**: ~3-5 minutes (downloads 5 images from 5 chapters with max 3 concurrent)
|
||||
|
||||
Times vary based on:
|
||||
- Network speed to source website
|
||||
- VPS performance
|
||||
- Current load on source website
|
||||
|
||||
## Contributing
|
||||
|
||||
When adding new features:
|
||||
1. Add corresponding tests in `test-vps-flow.js`
|
||||
2. If feature involves concurrent operations, add tests in `test-concurrent-downloads.js`
|
||||
3. Update this README with new test coverage
|
||||
4. Ensure all tests pass before submitting
|
||||
|
||||
## License
|
||||
|
||||
Same as the main MangaReader project.
|
||||
Reference in New Issue
Block a user