# PDF Processing Module

Complete module for processing PDF documents with mathematical content. Extracts text, formulas, and exercises from PDFs focused on Linear Algebra topics.

## Features

### 📄 Text Extraction
- Extract raw text from PDF documents
- Preserve page structure and layout
- Detect and extract headers, tables, and lists
- Language detection (Spanish/English)
- Text normalization and sanitization

### 🧮 Formula Parsing
- Detect mathematical formulas in LaTeX format
- Support for:
  - Inline formulas
  - Display formulas
  - Numbered equations
  - Matrices and determinants
  - Vectors
  - Fractions, powers, roots
  - Summations and integrals
  - Greek letters and operators
- Convert text notation to LaTeX
- Formula dependency tracking
- Confidence scoring

### 📝 Exercise Detection
- Identify exercises, problems, and examples
- Detect numbered and lettered exercises
- Extract exercise statements and solutions
- Categorize by topic (vectors, matrices, systems, etc.)
- Difficulty assessment (basic, intermediate, advanced)
- Tag generation
- Subexercise organization
- Solution detection

### ⚙️ Asynchronous Processing
- Bull queue for job management
- Redis-backed job queue
- Parallel processing with configurable concurrency
- Job priorities and rate limiting
- Automatic retry on failure
- Progress tracking
- Batch processing support

## Installation

```bash
cd backend
npm install
```

The module is already configured in `package.json` with required dependencies:
- `pdf-parse`: PDF text extraction
- `bull`: Job queue
- `ioredis`: Redis client
- `uuid`: Unique identifier generation

## Configuration

Environment variables in `.env`:

```env
# Redis Configuration
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_PASSWORD=
REDIS_DB=0

# PDF Storage
PDF_STORAGE_PATH=./uploads/pdfs
THUMBNAIL_PATH=./uploads/thumbnails

# Worker Configuration
PDF_WORKER_CONCURRENCY=2
PDF_WORKER_RATE_LIMIT=5
```

## Usage

### Quick Start

```typescript
import { PDFService } from '@modules/pdf/pdf.service';
import { container } from 'tsyringe';
import { PrismaService } from '@shared/services/prisma.service';

// Get services
const pdfService = container.resolve(PDFService);
const prisma = container.resolve(PrismaService);

// Get a PDF from database
const pdf = await prisma.pDF.findUnique({
  where: { id: 'pdf-id-here' }
});

// Process the PDF
const result = await pdfService.processPDF(pdf, {
  extractText: true,
  extractFormulas: true,
  detectExercises: true,
  generateThumbnails: false,
  language: 'es'
});

console.log('Formulas:', result.content?.formulas.length);
console.log('Exercises:', result.content?.exercises.length);
```

### Asynchronous Processing

```typescript
import { PDFProcessorWorker } from '@workers/pdf-processor.worker';

const worker = container.resolve(PDFProcessorWorker);

// Start worker
await worker.start();

// Add processing job
const job = await worker.addProcessJob('pdf-id-here', {
  extractText: true,
  extractFormulas: true,
  detectExercises: true,
  language: 'es'
});

// Monitor job status
const status = await worker.getJobStatus(job.id!);
console.log('Job state:', status.state);

// Stop worker when done
await worker.stop();
```

### Batch Processing

```typescript
// Process multiple PDFs
const pdfIds = ['pdf-1', 'pdf-2', 'pdf-3'];
const batchJob = await worker.addBatchJob(pdfIds);

// Process all PDFs from a topic
const jobIds = await worker.processTopicPDFs('topic-id-here');
```

### Working with Formulas

```typescript
import { FormulaParserProcessor } from '@modules/pdf/processors';

const formulaParser = container.resolve(FormulaParserProcessor);

// Get formulas by type
const matrices = formulaParser.getFormulasByType(formulas, 'matrix');
const vectors = formulaParser.getFormulasByType(formulas, 'vector');

// Search formulas
const integrals = formulaParser.searchFormulas(formulas, 'integral');

// Get high-confidence formulas
const reliable = formulaParser.getHighConfidenceFormulas(formulas, 0.8);
```

### Working with Exercises

```typescript
import { ExerciseDetectorProcessor } from '@modules/pdf/processors';

const exerciseDetector = container.resolve(ExerciseDetectorProcessor);

// Get exercises by type
const problems = exerciseDetector.getExercisesByType(exercises, 'problem');

// Get by difficulty
const basic = exerciseDetector.getExercisesByDifficulty(exercises, 'basic');

// Get by topic
const vectors = exerciseDetector.getExercisesByTopic(exercises, 'vectors');

// Get exercises with solutions
const withSolutions = exerciseDetector.getExercisesWithSolutions(exercises);

// Get statistics
const stats = exerciseDetector.getExerciseStatistics(exercises);
```

## API Reference

### PDFService

#### Methods

- `processPDF(pdfData, options)`: Process a single PDF
- `processBatch(pdfList, options)`: Process multiple PDFs
- `searchText(pdfData, query)`: Search text within PDF
- `extractPages(pdfPath, pages)`: Extract specific pages
- `getPreview(pdfData, pages)`: Get PDF preview
- `validatePDF(filePath)`: Validate PDF file

### PDFProcessorWorker

#### Methods

- `start()`: Start the worker
- `stop()`: Stop the worker
- `addProcessJob(pdfId, options, priority)`: Add processing job
- `addBatchJob(pdfIds, options)`: Add batch job
- `getJobStatus(jobId)`: Get job status
- `getQueueStats()`: Get queue statistics
- `pauseQueue()`: Pause processing queue
- `resumeQueue()`: Resume processing queue
- `retryFailedJobs()`: Retry failed jobs
- `processTopicPDFs(topicId)`: Process all PDFs from a topic

### TextExtractorProcessor

#### Methods

- `extract(pdfDoc, options)`: Extract text from PDF
- `extractTextByPage(pdfDoc, options)`: Extract text by page
- `extractPageText(pdfDoc, pageNumber)`: Extract specific page
- `extractStructure(pages)`: Extract document structure
- `extractKeyTerms(text, maxTerms)`: Extract key terms
- `extractNumberedLists(pages)`: Extract numbered lists

### FormulaParserProcessor

#### Methods

- `parse(text, textByPage, options)`: Parse formulas from text
- `getFormulasByType(formulas, type)`: Filter formulas by type
- `getHighConfidenceFormulas(formulas, threshold)`: Get reliable formulas
- `searchFormulas(formulas, query)`: Search formulas

### ExerciseDetectorProcessor

#### Methods

- `detect(text, textByPage, options)`: Detect exercises in text
- `getExercisesByType(exercises, type)`: Filter exercises by type
- `getExercisesByDifficulty(exercises, difficulty)`: Filter by difficulty
- `getExercisesByTopic(exercises, topic)`: Filter by topic
- `searchExercises(exercises, query)`: Search exercises
- `getExercisesWithSolutions(exercises)`: Get exercises with solutions
- `getExerciseStatistics(exercises)`: Get exercise statistics

## Exercise Numbering Patterns

The module detects exercises using these patterns:

### Spanish
- `Ejercicio N`, `Ejer N`, `EJERCICIO N`
- `Problema N`, `Prob N`, `PROBLEMA N`
- `Ejemplo N`, `EJEMPLO N`
- `Demostración N`, `Demostr N`
- `Práctica N`, `PRÁCTICA N`
- `N.`, `N)` (numbered)
- `a.`, `a)`, `i.`, `i)` (lettered/roman)

### English
- `Exercise N`, `Ex N`, `EXERCISE N`
- `Problem N`, `Prob N`, `PROBLEM N`
- `Example N`, `Ex N`, `EXAMPLE N`
- `Proof N`, `PROOF N`
- `Question N`, `Q N`, `QUESTION N`

## Topics

The module categorizes content into these topics:

1. **Vectors** (Vectores)
   - Dot product, cross product
   - Norm, magnitude, direction
   - Components

2. **Matrices** (Matrices)
   - Determinants
   - Inverse, transpose, trace
   - Rank, Hessian, Jacobian

3. **Systems** (Sistemas)
   - Linear equations
   - Gauss-Jordan elimination
   - Compatible/incompatible systems

4. **Vector Spaces** (Espacios Vectoriales)
   - Basis, dimension
   - Linear combinations
   - Span, generating sets

5. **Linear Programming** (Programación Lineal)
   - Optimization
   - Constraints, objective function
   - Simplex method, duality

## Examples

See `EXAMPLES.ts` for comprehensive usage examples including:

1. Process single PDF synchronously
2. Process PDFs asynchronously using worker
3. Process all PDFs from a topic
4. Search within PDFs
5. Extract specific pages
6. Get PDF preview
7. Work with formulas
8. Work with exercises
9. Batch process all PDFs
10. Retry failed jobs

## Performance

- **Processing Speed**: ~2-5 seconds per page (depends on content complexity)
- **Memory Usage**: ~50-100MB per processing job
- **Concurrency**: Configurable (default: 2 parallel jobs)
- **Rate Limiting**: 5 jobs per minute (configurable)

## Error Handling

The module includes comprehensive error handling:

- PDF validation before processing
- Automatic retry on transient failures
- Detailed error logging
- Graceful degradation
- Job failure tracking

## Testing

Run the module initialization script:

```bash
cd backend
npm run dev
# In another terminal:
npx tsx src/modules/pdf/pdf.init.ts
```

## Troubleshooting

### Common Issues

1. **PDF processing fails**
   - Check PDF is not corrupted
   - Verify PDF contains text (not just images)
   - Ensure sufficient disk space

2. **Worker won't start**
   - Check Redis is running
   - Verify Redis connection settings
   - Check port availability

3. **Low formula detection**
   - PDF may use images for formulas
   - Consider enabling OCR
   - Check formula patterns match your content

4. **Exercise detection misses items**
   - Verify exercise numbering patterns
   - Check language setting (es/en)
   - Review detection logs

## Contributing

To extend the module:

1. Add new processors in `processors/` directory
2. Register in DI container in `pdf.di.ts`
3. Export from `index.ts`
4. Add examples in `EXAMPLES.ts`
5. Update this README

## License

MIT License - See LICENSE file for details
